[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:03:44 dist-test-slave-lmfp krb5kdc[4042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:03:44 dist-test-slave-lmfp krb5kdc[4042](info): set up 2 sockets
Sep 27 23:03:44 dist-test-slave-lmfp krb5kdc[4042](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260927 23:03:45.520021 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:03:45 dist-test-slave-lmfp krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:03:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:03:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:03:45.682942 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38283
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:45.983754 4055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:45.984025 4055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:45.984110 4055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:45.991526 4055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:03:45.991626 4055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:45.991670 4055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:45.991725 4055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:03:45.991768 4055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:03:46.001368 4055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:46.002611 4055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:46.003952 4055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:46.015941 4061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:46.016032 4060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:46.016027 4063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:46.020491 4055 server_base.cc:1034] running on GCE node
I20260927 23:03:46.021448 4055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:46.022938 4055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:46.024183 4055 hybrid_clock.cc:648] HybridClock initialized: now 1790550226024150 us; error 35 us; skew 500 ppm
I20260927 23:03:46.024713 4055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:46 dist-test-slave-lmfp krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:46.032950 4055 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:03:46.034543 4055 webserver.cc:460] Webserver started at http://127.3.240.254:37909/ using document root <none> and password file <none>
I20260927 23:03:46.035368 4055 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:46.035518 4055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:46.035813 4055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:46.039143 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "2e2971076c84439290a480251e4d2e78"
format_stamp: "Formatted at 2026-09-27 23:03:46 on dist-test-slave-lmfp"
server_key: "d8ceb5d7c8853b995be9c85b9c19eb55"
server_key_iv: "b49771f4adc35ad235a93c6a94354305"
server_key_version: "encryptionkey@0"
I20260927 23:03:46.039883 4055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2e2971076c84439290a480251e4d2e78"
format_stamp: "Formatted at 2026-09-27 23:03:46 on dist-test-slave-lmfp"
server_key: "d8ceb5d7c8853b995be9c85b9c19eb55"
server_key_iv: "b49771f4adc35ad235a93c6a94354305"
server_key_version: "encryptionkey@0"
I20260927 23:03:46.044783 4055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 23:03:46.047991 4070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:46.049170 4055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:03:46.049341 4055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "2e2971076c84439290a480251e4d2e78"
format_stamp: "Formatted at 2026-09-27 23:03:46 on dist-test-slave-lmfp"
server_key: "d8ceb5d7c8853b995be9c85b9c19eb55"
server_key_iv: "b49771f4adc35ad235a93c6a94354305"
server_key_version: "encryptionkey@0"
I20260927 23:03:46.049546 4055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:46.082352 4055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:46.086185 4055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:46.086462 4055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:46.104163 4055 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:38283
I20260927 23:03:46.104193 4121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:38283 every 8 connection(s)
I20260927 23:03:46.105423 4055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:03:46.107347 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 4055
I20260927 23:03:46.107656 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:03:46.109812 4122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:46.110054 4035 external_mini_cluster.cc:1428] Setting key f2e49ffde2af11b371c3e271b633c17f
I20260927 23:03:46.123840 4122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78: Bootstrap starting.
I20260927 23:03:46.127570 4122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:46.128618 4122 log.cc:824] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:46.132005 4122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78: No bootstrap required, opened a new log
I20260927 23:03:46.140717 4122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2971076c84439290a480251e4d2e78" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38283 } }
I20260927 23:03:46.141057 4122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:46.141168 4122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2971076c84439290a480251e4d2e78, State: Initialized, Role: FOLLOWER
Sep 27 23:03:46 dist-test-slave-lmfp krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:46.141799 4122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2971076c84439290a480251e4d2e78" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38283 } }
I20260927 23:03:46.142072 4122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:03:46.142204 4122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:03:46.142377 4122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:46.144649 4122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2971076c84439290a480251e4d2e78" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38283 } }
I20260927 23:03:46.145136 4122 leader_election.cc:302] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2971076c84439290a480251e4d2e78; no voters:
I20260927 23:03:46.145798 4122 leader_election.cc:288] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:03:46.145908 4127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:46.147068 4127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [term 1 LEADER]: Becoming Leader. State: Replica: 2e2971076c84439290a480251e4d2e78, State: Running, Role: LEADER
I20260927 23:03:46.147579 4127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2971076c84439290a480251e4d2e78" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38283 } }
I20260927 23:03:46.148033 4122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:03:46.151187 4129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2971076c84439290a480251e4d2e78. Latest consensus state: current_term: 1 leader_uuid: "2e2971076c84439290a480251e4d2e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2971076c84439290a480251e4d2e78" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38283 } } }
I20260927 23:03:46.151479 4129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [sys.catalog]: This master's current role is: LEADER
I20260927 23:03:46.150642 4128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e2971076c84439290a480251e4d2e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2971076c84439290a480251e4d2e78" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38283 } } }
I20260927 23:03:46.151914 4128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78 [sys.catalog]: This master's current role is: LEADER
I20260927 23:03:46.158154 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:46.114664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58754 (local address 127.3.240.254:38283)
0927 23:03:46.115708 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:46.115740 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:46.116858 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 23:03:46.128351 (+ 11493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:46.128375 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:46.128825 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:46.129208 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:46.132855 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:46.134177 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:46.136491 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:46.137145 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:46.137285 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:46.143215 (+ 5930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:46.143223 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:46.143235 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:46.143270 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:46.146757 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:46.147669 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:46.147676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:46.147686 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:46.147884 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:46.148380 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:46.148385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:46.148388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:46.148883 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:46.155335 (+ 6452us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:46.155682 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 23:03:46.156178 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":740,"thread_start_us":419,"threads_started":1}
I20260927 23:03:46.166087 4132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:03:46.171662 4132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:03:46.181326 4132 catalog_manager.cc:1347] Generated new cluster ID: ceea18c261b345e7a088a58202ed1217
I20260927 23:03:46.181444 4132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:03:46.201514 4132 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:03:46.202970 4132 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:03:46.214022 4132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e2971076c84439290a480251e4d2e78: Generated new TSK 0
I20260927 23:03:46.214709 4132 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:03:46.267242 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38283
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:46.588563 4148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:46.588830 4148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:46.588992 4148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:46.596154 4148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:46.596252 4148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:46.596475 4148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:03:46.606386 4148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:46.607682 4148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:46.609058 4148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:46.621682 4153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:46.621626 4154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:46.621716 4156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:46.622270 4148 server_base.cc:1034] running on GCE node
I20260927 23:03:46.622826 4148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:46.623672 4148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:46.624858 4148 hybrid_clock.cc:648] HybridClock initialized: now 1790550226624805 us; error 46 us; skew 500 ppm
I20260927 23:03:46.625146 4148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:46 dist-test-slave-lmfp krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:46.628191 4148 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:03:46.629611 4148 webserver.cc:460] Webserver started at http://127.3.240.193:38379/ using document root <none> and password file <none>
I20260927 23:03:46.630353 4148 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:46.630479 4148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:46.630780 4148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:46.633724 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "24edd588f9c943c59e7f787061c84575"
format_stamp: "Formatted at 2026-09-27 23:03:46 on dist-test-slave-lmfp"
server_key: "188a049f3701247563e84de901196ff9"
server_key_iv: "29ca94c308768d74dd38dc47be534b3d"
server_key_version: "encryptionkey@0"
I20260927 23:03:46.634440 4148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "24edd588f9c943c59e7f787061c84575"
format_stamp: "Formatted at 2026-09-27 23:03:46 on dist-test-slave-lmfp"
server_key: "188a049f3701247563e84de901196ff9"
server_key_iv: "29ca94c308768d74dd38dc47be534b3d"
server_key_version: "encryptionkey@0"
I20260927 23:03:46.639259 4148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260927 23:03:46.642493 4163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:46.643762 4148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:03:46.643927 4148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "24edd588f9c943c59e7f787061c84575"
format_stamp: "Formatted at 2026-09-27 23:03:46 on dist-test-slave-lmfp"
server_key: "188a049f3701247563e84de901196ff9"
server_key_iv: "29ca94c308768d74dd38dc47be534b3d"
server_key_version: "encryptionkey@0"
I20260927 23:03:46.644120 4148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:46.660619 4148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:46.664786 4148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:46.665057 4148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:46.665910 4148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:03:46.667259 4148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:46.667358 4148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:46.667523 4148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:46.667594 4148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:46.698902 4148 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:42469
I20260927 23:03:46.698944 4275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:42469 every 8 connection(s)
I20260927 23:03:46.700177 4148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:03:46.703189 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 4148
I20260927 23:03:46.703410 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:03:46.704218 4035 external_mini_cluster.cc:1428] Setting key 32a02eb51d2b0e5f49c267c32b3345d3
Sep 27 23:03:46 dist-test-slave-lmfp krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550226, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260927 23:03:46.721942 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:46.702773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57581 (local address 127.3.240.254:38283)
0927 23:03:46.703384 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:46.703393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:46.704465 (+ 1072us) server_negotiation.cc:407] Connection header received
0927 23:03:46.706418 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:46.706422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:46.706500 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:46.706621 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:46.708108 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:46.708807 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:46.710868 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:46.711649 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:46.711783 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:46.715095 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:46.715104 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:46.715111 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:46.715153 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:46.718692 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:46.719791 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:46.719796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:46.719800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:46.719944 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:46.720391 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:46.720394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:46.720396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:46.720638 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:46.720741 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:46.721546 (+ 805us) server_negotiation.cc:299] Negotiation successful
0927 23:03:46.721708 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":384,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260927 23:03:46.723332 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:46.703513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38283 (local address 127.3.240.193:57581)
0927 23:03:46.704267 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:46.704347 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:46.706178 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:46.706882 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:46.706890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:46.707425 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:46.707953 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:46.707969 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:46.708979 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:46.708984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:46.710714 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:46.710731 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:46.711865 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:46.711870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:46.712083 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:46.712767 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:46.712787 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:46.714855 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:46.718850 (+ 3995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:46.718857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:46.718869 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:46.719671 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:46.720084 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:46.720088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:46.720090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:46.720259 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:46.720751 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:46.720756 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:46.721224 (+ 468us) client_negotiation.cc:769] Sending connection context
0927 23:03:46.721454 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 23:03:46.721937 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":317,"threads_started":1}
I20260927 23:03:46.724903 4276 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38283
I20260927 23:03:46.725240 4276 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:46.725956 4276 heartbeater.cc:511] Master 127.3.240.254:38283 requested a full tablet report, sending...
I20260927 23:03:46.728041 4087 ts_manager.cc:194] Registered new tserver with Master: 24edd588f9c943c59e7f787061c84575 (127.3.240.193:42469)
I20260927 23:03:46.729426 4087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:57581
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:03:46.741889 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38283
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:47.048058 4282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:47.048452 4282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:47.048640 4282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:47.057009 4282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:47.057175 4282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:47.057379 4282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:03:47.068379 4282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:47.069726 4282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:47.071204 4282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:47.084415 4288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:47.084354 4287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:47.084573 4290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:47.085242 4282 server_base.cc:1034] running on GCE node
I20260927 23:03:47.085757 4282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:47.086596 4282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:47.087781 4282 hybrid_clock.cc:648] HybridClock initialized: now 1790550227087752 us; error 32 us; skew 500 ppm
I20260927 23:03:47.088047 4282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:47.090914 4282 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:03:47.092334 4282 webserver.cc:460] Webserver started at http://127.3.240.194:33537/ using document root <none> and password file <none>
I20260927 23:03:47.092927 4282 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:47.093048 4282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:47.093295 4282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:47.096004 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "e663b3f54b29454d8b575855b293de12"
format_stamp: "Formatted at 2026-09-27 23:03:47 on dist-test-slave-lmfp"
server_key: "f7d77410a19b114c2d4416e1baac226c"
server_key_iv: "4e4aeaba2d9a571a22e2c04231b35e5e"
server_key_version: "encryptionkey@0"
I20260927 23:03:47.096663 4282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "e663b3f54b29454d8b575855b293de12"
format_stamp: "Formatted at 2026-09-27 23:03:47 on dist-test-slave-lmfp"
server_key: "f7d77410a19b114c2d4416e1baac226c"
server_key_iv: "4e4aeaba2d9a571a22e2c04231b35e5e"
server_key_version: "encryptionkey@0"
I20260927 23:03:47.101369 4282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:03:47.106352 4297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:47.108141 4282 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260927 23:03:47.108292 4282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "e663b3f54b29454d8b575855b293de12"
format_stamp: "Formatted at 2026-09-27 23:03:47 on dist-test-slave-lmfp"
server_key: "f7d77410a19b114c2d4416e1baac226c"
server_key_iv: "4e4aeaba2d9a571a22e2c04231b35e5e"
server_key_version: "encryptionkey@0"
I20260927 23:03:47.108533 4282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:47.136371 4282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:47.140199 4282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:47.140471 4282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:47.141378 4282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:03:47.142704 4282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:47.142802 4282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:47.142939 4282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:47.143036 4282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 23:03:47.172937 4282 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:45735
I20260927 23:03:47.172972 4409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:45735 every 8 connection(s)
I20260927 23:03:47.174113 4282 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:03:47.176057 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 4282
I20260927 23:03:47.176242 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:03:47.176734 4035 external_mini_cluster.cc:1428] Setting key ddfd5e3a8bb13b66076e3ccb90860846
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
I20260927 23:03:47.196372 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.176583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:48363 (local address 127.3.240.254:38283)
0927 23:03:47.176781 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:47.176787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:47.178145 (+ 1358us) server_negotiation.cc:407] Connection header received
0927 23:03:47.180839 (+ 2694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:47.180843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:47.180899 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:47.180972 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:47.182932 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.183659 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.185445 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.186195 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.186330 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.189831 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:47.189838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:47.189843 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:47.189889 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:47.193579 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.194293 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.194297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.194299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.194414 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.194787 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.194790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.194792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.195087 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:47.195198 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:47.195909 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 23:03:47.196065 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:03:47.197435 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.177163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38283 (local address 127.3.240.194:48363)
0927 23:03:47.178007 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:47.178047 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:47.180549 (+ 2502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:47.181248 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:47.181263 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:47.182013 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:47.182773 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:47.182801 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.183806 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.183812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:47.185302 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:47.185318 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.186379 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.186383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:47.186566 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.187280 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:47.187301 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:47.189608 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:47.193719 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:47.193726 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:47.193738 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:47.194191 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:47.194513 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:47.194516 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:47.194518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:47.194673 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:47.195206 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:47.195213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:47.195524 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 23:03:47.195823 (+ 299us) client_negotiation.cc:240] Negotiation successful
0927 23:03:47.196273 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":325,"threads_started":1}
I20260927 23:03:47.198947 4410 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38283
I20260927 23:03:47.199280 4410 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:47.200142 4410 heartbeater.cc:511] Master 127.3.240.254:38283 requested a full tablet report, sending...
I20260927 23:03:47.201655 4087 ts_manager.cc:194] Registered new tserver with Master: e663b3f54b29454d8b575855b293de12 (127.3.240.194:45735)
I20260927 23:03:47.202454 4087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:48363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:03:47.217775 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38283
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:47.530246 4415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:47.530535 4415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:47.530722 4415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:47.538209 4415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:47.538334 4415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:47.538559 4415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:03:47.548748 4415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38283
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:47.550311 4415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:47.552335 4415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:47.565941 4421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:47.566068 4423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:47.566107 4415 server_base.cc:1034] running on GCE node
W20260927 23:03:47.566272 4420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:47.567096 4415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:47.568073 4415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:47.569295 4415 hybrid_clock.cc:648] HybridClock initialized: now 1790550227569255 us; error 51 us; skew 500 ppm
I20260927 23:03:47.569587 4415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:47.572472 4415 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:03:47.573700 4415 webserver.cc:460] Webserver started at http://127.3.240.195:42821/ using document root <none> and password file <none>
I20260927 23:03:47.574272 4415 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:47.574362 4415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:47.574595 4415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:47.577194 4415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "1d4b43e40be44799a837f305ef2ab563"
format_stamp: "Formatted at 2026-09-27 23:03:47 on dist-test-slave-lmfp"
server_key: "14ab99e844f0b45667b16a5e6d547139"
server_key_iv: "a89a12bf751de8bd98f9dc398a40e4a8"
server_key_version: "encryptionkey@0"
I20260927 23:03:47.577881 4415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "1d4b43e40be44799a837f305ef2ab563"
format_stamp: "Formatted at 2026-09-27 23:03:47 on dist-test-slave-lmfp"
server_key: "14ab99e844f0b45667b16a5e6d547139"
server_key_iv: "a89a12bf751de8bd98f9dc398a40e4a8"
server_key_version: "encryptionkey@0"
I20260927 23:03:47.582569 4415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260927 23:03:47.585649 4430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:47.586794 4415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:03:47.586954 4415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "1d4b43e40be44799a837f305ef2ab563"
format_stamp: "Formatted at 2026-09-27 23:03:47 on dist-test-slave-lmfp"
server_key: "14ab99e844f0b45667b16a5e6d547139"
server_key_iv: "a89a12bf751de8bd98f9dc398a40e4a8"
server_key_version: "encryptionkey@0"
I20260927 23:03:47.587181 4415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:47.606401 4415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:47.610216 4415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:47.610478 4415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:47.611467 4415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:03:47.612747 4415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:47.612814 4415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:47.612957 4415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:47.613013 4415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:47.644853 4415 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:36381
I20260927 23:03:47.644903 4542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:36381 every 8 connection(s)
I20260927 23:03:47.646087 4415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:03:47.651865 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 4415
I20260927 23:03:47.652062 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:03:47.652462 4035 external_mini_cluster.cc:1428] Setting key 3e81b3c26eda9e7c4d9b4074477e5b13
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:47.664592 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.648581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:44777 (local address 127.3.240.254:38283)
0927 23:03:47.648757 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:47.648764 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:47.650006 (+ 1242us) server_negotiation.cc:407] Connection header received
0927 23:03:47.651771 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:47.651776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:47.651870 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:47.652010 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:47.653687 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.654806 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.656439 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.656808 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.656897 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.659912 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:47.659917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:47.659920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:47.659948 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:47.662163 (+ 2215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.662774 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.662778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.662780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.662884 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.663292 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.663295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.663297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.663548 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:47.663658 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:47.664209 (+ 551us) server_negotiation.cc:299] Negotiation successful
0927 23:03:47.664357 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:03:47.665478 4545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.649137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38283 (local address 127.3.240.195:44777)
0927 23:03:47.649826 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:47.649881 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:47.651499 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:47.652298 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:47.652305 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:47.652939 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:47.653488 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:47.653503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.654964 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.654970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:47.656321 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:47.656331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.656973 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.656977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:47.657181 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.657740 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:47.657764 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:47.659752 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:47.662272 (+ 2520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:47.662280 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:47.662291 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:47.662667 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:47.663025 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:47.663032 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:47.663040 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:47.663189 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:47.663645 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:47.663651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:47.663900 (+ 249us) client_negotiation.cc:769] Sending connection context
0927 23:03:47.664127 (+ 227us) client_negotiation.cc:240] Negotiation successful
0927 23:03:47.664480 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":263,"threads_started":1}
I20260927 23:03:47.666884 4543 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38283
I20260927 23:03:47.667220 4543 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:47.668062 4543 heartbeater.cc:511] Master 127.3.240.254:38283 requested a full tablet report, sending...
I20260927 23:03:47.669304 4087 ts_manager.cc:194] Registered new tserver with Master: 1d4b43e40be44799a837f305ef2ab563 (127.3.240.195:36381)
I20260927 23:03:47.670037 4087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:44777
I20260927 23:03:47.679934 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:03:47.723290 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.708827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.3.240.193:42469)
0927 23:03:47.709465 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:47.709474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:47.709562 (+ 88us) server_negotiation.cc:407] Connection header received
0927 23:03:47.709766 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:47.709773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:47.709959 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:47.710118 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:47.711566 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.712522 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.713829 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.714421 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.714576 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.718242 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:47.718252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:47.718263 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:47.718291 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:47.720804 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.721397 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.721405 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.721407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.721526 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.721934 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.721938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.721941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.722271 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:47.722482 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:47.722750 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 23:03:47.722925 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":334,"threads_started":1}
W20260927 23:03:47.724445 4191 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50014
I20260927 23:03:47.732915 4276 heartbeater.cc:503] Master 127.3.240.254:38283 was elected leader, sending a full tablet report...
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:03:47.763924 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.750960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.3.240.193:42469)
0927 23:03:47.751209 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:47.751217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:47.751503 (+ 286us) server_negotiation.cc:407] Connection header received
0927 23:03:47.751691 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:47.751696 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:47.751763 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:47.751896 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:47.753151 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.753824 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.755006 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.755402 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.755532 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.759184 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:47.759188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:47.759192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:47.759240 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:47.761571 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.762141 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.762152 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.762161 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.762266 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.762629 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.762635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.762639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.762872 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:47.763124 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:47.763420 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 23:03:47.763640 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260927 23:03:47.765154 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 4148
I20260927 23:03:47.776182 4550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.766009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.3.240.193:42469)
0927 23:03:47.766144 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:47.766147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:47.766391 (+ 244us) server_negotiation.cc:407] Connection header received
0927 23:03:47.766481 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:47.766485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:47.766531 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:47.766600 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:47.767786 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.768517 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.769616 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.770006 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.770090 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.772108 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:47.772112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:47.772114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:47.772137 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:47.774275 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.774769 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.774772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.774774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.774875 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.775228 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.775231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.775233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.775438 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:47.775592 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:47.775801 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 23:03:47.775919 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:03:47.776794 4271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:03:47.856283 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 4148
I20260927 23:03:47.872835 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 4282
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:03:47.886796 4557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.873583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.3.240.194:45735)
0927 23:03:47.874158 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:47.874165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:47.874201 (+ 36us) server_negotiation.cc:407] Connection header received
0927 23:03:47.874255 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:47.874261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:47.874455 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:47.874593 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:47.875827 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.876791 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.877894 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.878416 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.878508 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.882162 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:47.882172 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:47.882186 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:47.882213 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:47.884747 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.885264 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.885270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.885272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.885369 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.885684 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.885687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.885689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.885959 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:47.886189 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:47.886378 (+ 189us) server_negotiation.cc:299] Negotiation successful
0927 23:03:47.886542 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":293,"threads_started":1}
I20260927 23:03:47.888149 4405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:03:47.968410 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 4282
I20260927 23:03:47.983958 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 4415
Sep 27 23:03:47 dist-test-slave-lmfp krb5kdc[4042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:03:47.998602 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:47.985045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53896 (local address 127.3.240.195:36381)
0927 23:03:47.985659 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:47.985666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:47.985692 (+ 26us) server_negotiation.cc:407] Connection header received
0927 23:03:47.985747 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:47.985754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:47.985960 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:47.986077 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:47.987213 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.988147 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.989326 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:47.989846 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:47.989966 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:47.993750 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:47.993760 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:47.993773 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:47.993805 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:47.996374 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.996947 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.996954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.996956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.997063 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:47.997340 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:47.997343 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:47.997345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:47.997642 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:47.997843 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:47.998068 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 23:03:47.998241 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":339,"threads_started":1}
I20260927 23:03:47.999938 4538 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:03:48.080163 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 4415
I20260927 23:03:48.095173 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 4055
I20260927 23:03:48.096360 4117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:03:48.196260 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 4055
2026-09-27T23:03:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3732 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:03:48 dist-test-slave-lmfp krb5kdc[4564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:03:48 dist-test-slave-lmfp krb5kdc[4564](info): set up 2 sockets
Sep 27 23:03:48 dist-test-slave-lmfp krb5kdc[4564](info): commencing operation
krb5kdc: starting...
W20260927 23:03:49.234735 4035 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.005s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:03:49 dist-test-slave-lmfp krb5kdc[4564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:03:49.392349 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39491
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:49.707772 4577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:49.708046 4577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:49.708220 4577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:49.715178 4577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:03:49.715260 4577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:49.715346 4577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:49.715431 4577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:03:49.715512 4577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:03:49.724452 4577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:49.725716 4577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:49.726964 4577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:49.738116 4583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:49.738178 4585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:49.738149 4582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:49.738875 4577 server_base.cc:1034] running on GCE node
I20260927 23:03:49.739566 4577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:49.740828 4577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:49.742038 4577 hybrid_clock.cc:648] HybridClock initialized: now 1790550229742000 us; error 56 us; skew 500 ppm
I20260927 23:03:49.742349 4577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:49 dist-test-slave-lmfp krb5kdc[4564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550229, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:49.745225 4577 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:03:49.746624 4577 webserver.cc:460] Webserver started at http://127.3.240.254:36557/ using document root <none> and password file <none>
I20260927 23:03:49.747318 4577 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:49.747449 4577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:49.747756 4577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:49.750391 4577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "9daf855e8d3f4ff69dfe2d20608c34fc"
format_stamp: "Formatted at 2026-09-27 23:03:49 on dist-test-slave-lmfp"
server_key: "01ccd69a0c5e603a9a8637d947df9066"
server_key_iv: "9e5518a7ce699738888d758e64df71b8"
server_key_version: "encryptionkey@0"
I20260927 23:03:49.751183 4577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "9daf855e8d3f4ff69dfe2d20608c34fc"
format_stamp: "Formatted at 2026-09-27 23:03:49 on dist-test-slave-lmfp"
server_key: "01ccd69a0c5e603a9a8637d947df9066"
server_key_iv: "9e5518a7ce699738888d758e64df71b8"
server_key_version: "encryptionkey@0"
I20260927 23:03:49.755688 4577 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 23:03:49.758682 4592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:49.759859 4577 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:03:49.760008 4577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "9daf855e8d3f4ff69dfe2d20608c34fc"
format_stamp: "Formatted at 2026-09-27 23:03:49 on dist-test-slave-lmfp"
server_key: "01ccd69a0c5e603a9a8637d947df9066"
server_key_iv: "9e5518a7ce699738888d758e64df71b8"
server_key_version: "encryptionkey@0"
I20260927 23:03:49.760196 4577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:49.773104 4577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:49.777029 4577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:49.777357 4577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:49.794353 4577 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39491
I20260927 23:03:49.794425 4643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39491 every 8 connection(s)
I20260927 23:03:49.795562 4577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:03:49.798832 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:49.805698 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 4577
I20260927 23:03:49.805928 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:03:49.806669 4035 external_mini_cluster.cc:1428] Setting key 2be6fcb026744a10b0ac1df36df5ba4c
I20260927 23:03:49.808888 4644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc: Bootstrap starting.
I20260927 23:03:49.811934 4644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:49.813215 4644 log.cc:824] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:49.816218 4644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc: No bootstrap required, opened a new log
Sep 27 23:03:49 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:49.824527 4644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daf855e8d3f4ff69dfe2d20608c34fc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39491 } }
I20260927 23:03:49.824922 4644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:49.825052 4644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9daf855e8d3f4ff69dfe2d20608c34fc, State: Initialized, Role: FOLLOWER
I20260927 23:03:49.825649 4644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daf855e8d3f4ff69dfe2d20608c34fc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39491 } }
I20260927 23:03:49.825886 4644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:03:49.826016 4644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:03:49.826197 4644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:49.828598 4644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daf855e8d3f4ff69dfe2d20608c34fc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39491 } }
I20260927 23:03:49.828817 4647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:49.808676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.3.240.254:39491)
0927 23:03:49.809376 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:49.809387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:49.809424 (+ 37us) server_negotiation.cc:407] Connection header received
0927 23:03:49.811066 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:49.811099 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:49.811453 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:49.811738 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:49.813491 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:49.814800 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:49.816498 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:49.817123 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:49.817260 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:49.821557 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:49.821564 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:49.821583 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:49.821636 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:49.824757 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:49.825501 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:49.825513 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:49.825518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:49.825680 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:49.826061 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:49.826066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:49.826069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:49.826664 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:49.826977 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:49.827485 (+ 508us) server_negotiation.cc:299] Negotiation successful
0927 23:03:49.827886 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":336,"threads_started":1}
I20260927 23:03:49.829125 4644 leader_election.cc:302] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9daf855e8d3f4ff69dfe2d20608c34fc; no voters:
I20260927 23:03:49.829734 4644 leader_election.cc:288] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:03:49.829851 4649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:49.831171 4649 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [term 1 LEADER]: Becoming Leader. State: Replica: 9daf855e8d3f4ff69dfe2d20608c34fc, State: Running, Role: LEADER
I20260927 23:03:49.831797 4649 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daf855e8d3f4ff69dfe2d20608c34fc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39491 } }
I20260927 23:03:49.832165 4644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:03:49.834563 4650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9daf855e8d3f4ff69dfe2d20608c34fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daf855e8d3f4ff69dfe2d20608c34fc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39491 } } }
I20260927 23:03:49.834540 4651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9daf855e8d3f4ff69dfe2d20608c34fc. Latest consensus state: current_term: 1 leader_uuid: "9daf855e8d3f4ff69dfe2d20608c34fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9daf855e8d3f4ff69dfe2d20608c34fc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39491 } } }
I20260927 23:03:49.835207 4650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [sys.catalog]: This master's current role is: LEADER
I20260927 23:03:49.835212 4651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc [sys.catalog]: This master's current role is: LEADER
I20260927 23:03:49.837531 4654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:03:49.842402 4654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:03:49.849706 4654 catalog_manager.cc:1347] Generated new cluster ID: e1e0599550674fdd9519b4e449b7ee0f
I20260927 23:03:49.849797 4654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:03:49.870461 4654 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:03:49.871716 4654 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:03:49.878911 4654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9daf855e8d3f4ff69dfe2d20608c34fc: Generated new TSK 0
I20260927 23:03:49.879670 4654 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:03:49.920414 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39491
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:50.225189 4670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:50.225436 4670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:50.225526 4670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:50.232471 4670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:50.232586 4670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:50.232692 4670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:03:50.242126 4670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39491
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:50.243265 4670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:50.244513 4670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:50.256543 4676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:50.256579 4675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:50.256673 4678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:50.257465 4670 server_base.cc:1034] running on GCE node
I20260927 23:03:50.258041 4670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:50.258896 4670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:50.260107 4670 hybrid_clock.cc:648] HybridClock initialized: now 1790550230260071 us; error 65 us; skew 500 ppm
I20260927 23:03:50.260412 4670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:50 dist-test-slave-lmfp krb5kdc[4564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550230, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:50.263576 4670 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:03:50.264966 4670 webserver.cc:460] Webserver started at http://127.3.240.193:37127/ using document root <none> and password file <none>
I20260927 23:03:50.265528 4670 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:50.265619 4670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:50.265854 4670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:50.268429 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "05af018fe6184300a8f4dce841bf4e20"
format_stamp: "Formatted at 2026-09-27 23:03:50 on dist-test-slave-lmfp"
server_key: "b93bc1a4a5d5aba155726cad4dd515b3"
server_key_iv: "271326b3599095a0f813fbe37f45ffc5"
server_key_version: "encryptionkey@0"
I20260927 23:03:50.269052 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "05af018fe6184300a8f4dce841bf4e20"
format_stamp: "Formatted at 2026-09-27 23:03:50 on dist-test-slave-lmfp"
server_key: "b93bc1a4a5d5aba155726cad4dd515b3"
server_key_iv: "271326b3599095a0f813fbe37f45ffc5"
server_key_version: "encryptionkey@0"
I20260927 23:03:50.273432 4670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:03:50.276608 4685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:50.277755 4670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:03:50.277915 4670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "05af018fe6184300a8f4dce841bf4e20"
format_stamp: "Formatted at 2026-09-27 23:03:50 on dist-test-slave-lmfp"
server_key: "b93bc1a4a5d5aba155726cad4dd515b3"
server_key_iv: "271326b3599095a0f813fbe37f45ffc5"
server_key_version: "encryptionkey@0"
I20260927 23:03:50.278107 4670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:50.311877 4670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:50.316293 4670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:50.316573 4670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:50.317476 4670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:03:50.318710 4670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:50.318809 4670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:50.318951 4670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:50.319056 4670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260927 23:03:50.349874 4670 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:34919
I20260927 23:03:50.349934 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:34919 every 8 connection(s)
I20260927 23:03:50.351135 4670 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:03:50.355371 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 4670
I20260927 23:03:50.355587 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:03:50.356318 4035 external_mini_cluster.cc:1428] Setting key 9311eb8e8fff818b7f58468767ff3f99
Sep 27 23:03:50 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550230, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260927 23:03:50.373245 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:50.353564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:49531 (local address 127.3.240.254:39491)
0927 23:03:50.353964 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:50.353968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:50.354935 (+ 967us) server_negotiation.cc:407] Connection header received
0927 23:03:50.357467 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:50.357471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:50.357536 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:50.357649 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:50.359533 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:50.360213 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:50.362675 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:50.363405 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:50.363541 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:50.366838 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:50.366846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:50.366853 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:50.366895 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:50.370370 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:50.370959 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:50.370964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:50.370969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:50.371134 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:50.371521 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:50.371529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:50.371534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:50.371902 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:50.372043 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:50.372644 (+ 601us) server_negotiation.cc:299] Negotiation successful
0927 23:03:50.372882 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":209,"threads_started":1}
I20260927 23:03:50.373986 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:50.354158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39491 (local address 127.3.240.193:49531)
0927 23:03:50.354801 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:50.354841 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:50.357201 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:50.357847 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:50.357854 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:50.358583 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:50.359365 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:50.359392 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:50.360361 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:50.360369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:50.362488 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:50.362506 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:50.363596 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:50.363601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:50.363780 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:50.364456 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:50.364480 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:50.366629 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:50.370493 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:50.370498 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:50.370509 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:50.370855 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:50.371244 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:50.371247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:50.371249 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:50.371418 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:50.372025 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:50.372029 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:50.372299 (+ 270us) client_negotiation.cc:769] Sending connection context
0927 23:03:50.372504 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 23:03:50.372826 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":260,"threads_started":1}
I20260927 23:03:50.375463 4798 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39491
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260927 23:03:50.375766 4798 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:50.376425 4798 heartbeater.cc:511] Master 127.3.240.254:39491 requested a full tablet report, sending...
I20260927 23:03:50.378350 4609 ts_manager.cc:194] Registered new tserver with Master: 05af018fe6184300a8f4dce841bf4e20 (127.3.240.193:34919)
I20260927 23:03:50.380326 4609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:49531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:03:50.395485 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39491
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:50.697142 4804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:50.697432 4804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:50.697759 4804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:50.705041 4804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:50.705199 4804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:50.705375 4804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:03:50.714450 4804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39491
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:50.715725 4804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:50.717139 4804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:50.729810 4809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:50.729861 4810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:50.730410 4812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:50.730517 4804 server_base.cc:1034] running on GCE node
I20260927 23:03:50.731186 4804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:50.732120 4804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:50.733332 4804 hybrid_clock.cc:648] HybridClock initialized: now 1790550230733275 us; error 49 us; skew 500 ppm
I20260927 23:03:50.733721 4804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:50 dist-test-slave-lmfp krb5kdc[4564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550230, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:50.736800 4804 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:03:50.738287 4804 webserver.cc:460] Webserver started at http://127.3.240.194:46079/ using document root <none> and password file <none>
I20260927 23:03:50.738953 4804 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:50.739137 4804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:50.739447 4804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:50.742233 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "4f53119a191e42fbb2c6b8d71c523ddc"
format_stamp: "Formatted at 2026-09-27 23:03:50 on dist-test-slave-lmfp"
server_key: "6f79f1e6c7f3b8ca89fab4ccf7e62d58"
server_key_iv: "7fc6fc9ecd1c988b4f206ef7fe60b4ac"
server_key_version: "encryptionkey@0"
I20260927 23:03:50.743086 4804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "4f53119a191e42fbb2c6b8d71c523ddc"
format_stamp: "Formatted at 2026-09-27 23:03:50 on dist-test-slave-lmfp"
server_key: "6f79f1e6c7f3b8ca89fab4ccf7e62d58"
server_key_iv: "7fc6fc9ecd1c988b4f206ef7fe60b4ac"
server_key_version: "encryptionkey@0"
I20260927 23:03:50.747912 4804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 23:03:50.751195 4819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:50.752331 4804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:03:50.752525 4804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "4f53119a191e42fbb2c6b8d71c523ddc"
format_stamp: "Formatted at 2026-09-27 23:03:50 on dist-test-slave-lmfp"
server_key: "6f79f1e6c7f3b8ca89fab4ccf7e62d58"
server_key_iv: "7fc6fc9ecd1c988b4f206ef7fe60b4ac"
server_key_version: "encryptionkey@0"
I20260927 23:03:50.752729 4804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:50.768643 4804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:50.772842 4804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:50.773133 4804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:50.774006 4804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:03:50.775434 4804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:50.775556 4804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:50.775708 4804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:50.775789 4804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:50.809100 4804 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:42941
I20260927 23:03:50.809135 4931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:42941 every 8 connection(s)
I20260927 23:03:50.810300 4804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:03:50.819203 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 4804
I20260927 23:03:50.819401 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:03:50.819818 4035 external_mini_cluster.cc:1428] Setting key 4553dbccedd992e0a3d09ee6ddcc0772
Sep 27 23:03:50 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550230, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:50.829535 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:50.812772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52819 (local address 127.3.240.254:39491)
0927 23:03:50.812926 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:50.812930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:50.813988 (+ 1058us) server_negotiation.cc:407] Connection header received
0927 23:03:50.815630 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:50.815635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:50.815712 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:50.815807 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:50.817070 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:50.817729 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:50.819494 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:50.819925 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:50.820072 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:50.823718 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:50.823726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:50.823733 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:50.823775 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:50.826526 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:50.827469 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:50.827472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:50.827474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:50.827591 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:50.828045 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:50.828048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:50.828050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:50.828254 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:50.828351 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:50.829196 (+ 845us) server_negotiation.cc:299] Negotiation successful
0927 23:03:50.829336 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:03:50.831065 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:50.813204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39491 (local address 127.3.240.194:52819)
0927 23:03:50.813848 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:50.813885 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:50.815398 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:50.816015 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:50.816022 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:50.816448 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:50.816919 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:50.816932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:50.817870 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:50.817874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:50.819332 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:50.819345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:50.820169 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:50.820178 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:50.820494 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:50.821179 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:50.821214 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:50.823520 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:50.826660 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:50.826672 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:50.826689 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:50.827336 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:50.827693 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:50.827701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:50.827708 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:50.827940 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:50.828371 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:50.828382 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:50.828850 (+ 468us) client_negotiation.cc:769] Sending connection context
0927 23:03:50.829142 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 23:03:50.829672 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":281,"threads_started":1}
I20260927 23:03:50.832464 4932 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39491
I20260927 23:03:50.832736 4932 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:50.833496 4932 heartbeater.cc:511] Master 127.3.240.254:39491 requested a full tablet report, sending...
I20260927 23:03:50.834717 4609 ts_manager.cc:194] Registered new tserver with Master: 4f53119a191e42fbb2c6b8d71c523ddc (127.3.240.194:42941)
I20260927 23:03:50.835513 4609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:52819
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:03:50.856112 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39491
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:51.168745 4937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:51.168994 4937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:51.169129 4937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:51.175993 4937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:51.176105 4937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:51.176268 4937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:03:51.185367 4937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39491
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:51.186550 4937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:51.187883 4937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:51.200234 4942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:51.200273 4943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:51.200498 4945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:51.200770 4937 server_base.cc:1034] running on GCE node
I20260927 23:03:51.201352 4937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:51.202123 4937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:51.203318 4937 hybrid_clock.cc:648] HybridClock initialized: now 1790550231203278 us; error 49 us; skew 500 ppm
I20260927 23:03:51.203593 4937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:51.206463 4937 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:03:51.207798 4937 webserver.cc:460] Webserver started at http://127.3.240.195:43111/ using document root <none> and password file <none>
I20260927 23:03:51.208451 4937 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:51.208601 4937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:51.208871 4937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:51.211557 4937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "d8c09dc8a169463d809dc88fd3fea42c"
format_stamp: "Formatted at 2026-09-27 23:03:51 on dist-test-slave-lmfp"
server_key: "9a24c0bc99955b2105a2fa8f8408fe6a"
server_key_iv: "f409c322485135642830000f67619513"
server_key_version: "encryptionkey@0"
I20260927 23:03:51.212260 4937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "d8c09dc8a169463d809dc88fd3fea42c"
format_stamp: "Formatted at 2026-09-27 23:03:51 on dist-test-slave-lmfp"
server_key: "9a24c0bc99955b2105a2fa8f8408fe6a"
server_key_iv: "f409c322485135642830000f67619513"
server_key_version: "encryptionkey@0"
I20260927 23:03:51.216845 4937 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:03:51.219933 4952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:51.221062 4937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:03:51.221211 4937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "d8c09dc8a169463d809dc88fd3fea42c"
format_stamp: "Formatted at 2026-09-27 23:03:51 on dist-test-slave-lmfp"
server_key: "9a24c0bc99955b2105a2fa8f8408fe6a"
server_key_iv: "f409c322485135642830000f67619513"
server_key_version: "encryptionkey@0"
I20260927 23:03:51.221398 4937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:51.234174 4937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:51.238080 4937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:51.238348 4937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:51.239248 4937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:03:51.240356 4937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:51.240450 4937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:51.240585 4937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:51.240656 4937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:51.271245 4937 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:39201
I20260927 23:03:51.271283 5064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:39201 every 8 connection(s)
I20260927 23:03:51.272413 4937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:03:51.279767 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 4937
I20260927 23:03:51.279927 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:03:51.280378 4035 external_mini_cluster.cc:1428] Setting key b00eea96b3bf710b2f88d0a5ae22d440
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550231, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:51.290501 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.274797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47611 (local address 127.3.240.254:39491)
0927 23:03:51.274948 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.274953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.276274 (+ 1321us) server_negotiation.cc:407] Connection header received
0927 23:03:51.277850 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.277854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.277911 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.278012 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:51.279353 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.280066 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.281883 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.282368 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.282488 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.285645 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:51.285649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:51.285652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:51.285678 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:51.288030 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.288656 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.288660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.288662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.288785 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.289149 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.289152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.289154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.289401 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:51.289528 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.290061 (+ 533us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.290233 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:03:51.291286 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.275294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39491 (local address 127.3.240.195:47611)
0927 23:03:51.276098 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:51.276148 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:51.277649 (+ 1501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:51.278201 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:51.278208 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:51.278653 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:51.279217 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.279233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.280185 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.280190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.281747 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.281760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.282563 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.282571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.282855 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.283475 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:51.283502 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:51.285474 (+ 1972us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:51.288143 (+ 2669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:51.288148 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:51.288158 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:51.288512 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:51.288885 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:51.288888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:51.288890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:51.289045 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:51.289505 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:51.289509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:51.289767 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 23:03:51.289950 (+ 183us) client_negotiation.cc:240] Negotiation successful
0927 23:03:51.290284 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":365,"threads_started":1}
I20260927 23:03:51.292536 5065 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39491
I20260927 23:03:51.292795 5065 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:51.293403 5065 heartbeater.cc:511] Master 127.3.240.254:39491 requested a full tablet report, sending...
I20260927 23:03:51.294847 4609 ts_manager.cc:194] Registered new tserver with Master: d8c09dc8a169463d809dc88fd3fea42c (127.3.240.195:39201)
I20260927 23:03:51.295681 4609 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:47611
I20260927 23:03:51.307049 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:03:51.323237 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.312606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.3.240.254:39491)
0927 23:03:51.312765 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.312769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.313266 (+ 497us) server_negotiation.cc:407] Connection header received
0927 23:03:51.313385 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.313389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.313440 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.313521 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:51.314795 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.315518 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.316576 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.316933 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.317014 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.318897 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:51.318901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:51.318904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:51.318928 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:51.321139 (+ 2211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.321658 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.321661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.321663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.321763 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.322113 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.322115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.322117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.322370 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:51.322536 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.322847 (+ 311us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.322978 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 23:03:51.329030 4609 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:03:51.335716 4609 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:03:51.361757 5077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.346501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:39201 (local address 127.0.0.1:60112)
0927 23:03:51.347408 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:51.347449 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:51.347605 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:51.348587 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:51.348618 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:51.348664 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:51.349057 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.349073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.352731 (+ 3658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.352739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.355693 (+ 2954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.355712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.359469 (+ 3757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.359489 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.359782 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.359887 (+ 105us) client_negotiation.cc:769] Sending connection context
0927 23:03:51.360002 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 23:03:51.361332 (+ 1330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"mutex_wait_us":3,"thread_start_us":606,"threads_started":1}
I20260927 23:03:51.362234 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.346750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60112 (local address 127.3.240.195:39201)
0927 23:03:51.348006 (+ 1256us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.348016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.348048 (+ 32us) server_negotiation.cc:407] Connection header received
0927 23:03:51.348141 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.348148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.348371 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.348556 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:51.349340 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.352545 (+ 3205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.357185 (+ 4640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.359222 (+ 2037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.361455 (+ 2233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.361669 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.361746 (+ 77us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.361915 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":992,"threads_started":1}
I20260927 23:03:51.366012 5000 tablet_service.cc:1467] Processing CreateTablet for tablet 8454dbde82a344d08907758d422124cc (DEFAULT_TABLE table=test-table [id=3714bec2068a48f68467ff83b1672c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:51.367059 5078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.346730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:42941 (local address 127.0.0.1:59896)
0927 23:03:51.348628 (+ 1898us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:51.348663 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:51.348791 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:51.359526 (+ 10735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:51.359533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:51.359568 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:51.360359 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.360376 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.362348 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.362353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.364449 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.364462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.366265 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.366271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.366427 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.366455 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 23:03:51.366542 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 23:03:51.366658 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1773,"mutex_wait_us":249,"thread_start_us":1641,"threads_started":1}
I20260927 23:03:51.367096 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.347178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.3.240.194:42941)
0927 23:03:51.348253 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.348260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.348861 (+ 601us) server_negotiation.cc:407] Connection header received
0927 23:03:51.348959 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.348966 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.349171 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.359247 (+ 10076us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:51.360592 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.362196 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.364623 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.365974 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.366121 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.366292 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.366635 (+ 343us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.366784 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":857,"threads_started":1}
I20260927 23:03:51.367437 5000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8454dbde82a344d08907758d422124cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:51.368881 5081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.348763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34919 (local address 127.0.0.1:41336)
0927 23:03:51.361485 (+ 12722us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:51.361513 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:51.361612 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:51.362726 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:51.362736 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:51.362760 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:51.363170 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.363184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.364884 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.364889 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.366725 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.366737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.368160 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.368166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.368326 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.368362 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 23:03:51.368444 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 23:03:51.368566 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12597,"thread_start_us":12469,"threads_started":1}
I20260927 23:03:51.368932 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.349213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41336 (local address 127.3.240.193:34919)
0927 23:03:51.362038 (+ 12825us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.362063 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.362085 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:03:51.362163 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.362169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.362378 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.362583 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:51.363366 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.364766 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.366882 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.367966 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.368107 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.368309 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.368479 (+ 170us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.368616 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12693,"thread_start_us":13229,"threads_started":1}
I20260927 23:03:51.370334 4867 tablet_service.cc:1467] Processing CreateTablet for tablet 8454dbde82a344d08907758d422124cc (DEFAULT_TABLE table=test-table [id=3714bec2068a48f68467ff83b1672c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:51.371564 4867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8454dbde82a344d08907758d422124cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:51.371591 4733 tablet_service.cc:1467] Processing CreateTablet for tablet 8454dbde82a344d08907758d422124cc (DEFAULT_TABLE table=test-table [id=3714bec2068a48f68467ff83b1672c40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:51.372685 4733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8454dbde82a344d08907758d422124cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:51.381078 5083 tablet_bootstrap.cc:492] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c: Bootstrap starting.
I20260927 23:03:51.383262 4798 heartbeater.cc:503] Master 127.3.240.254:39491 was elected leader, sending a full tablet report...
I20260927 23:03:51.385388 5083 tablet_bootstrap.cc:654] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:51.387256 5083 log.cc:824] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:51.388543 5084 tablet_bootstrap.cc:492] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20: Bootstrap starting.
I20260927 23:03:51.388788 5085 tablet_bootstrap.cc:492] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc: Bootstrap starting.
I20260927 23:03:51.390290 5083 tablet_bootstrap.cc:492] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c: No bootstrap required, opened a new log
I20260927 23:03:51.390542 5083 ts_tablet_manager.cc:1397] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260927 23:03:51.392165 5085 tablet_bootstrap.cc:654] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:51.392750 5084 tablet_bootstrap.cc:654] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:51.393321 5085 log.cc:824] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:51.393838 5084 log.cc:824] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:51.396445 5084 tablet_bootstrap.cc:492] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20: No bootstrap required, opened a new log
I20260927 23:03:51.396723 5084 ts_tablet_manager.cc:1397] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 23:03:51.396765 5085 tablet_bootstrap.cc:492] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc: No bootstrap required, opened a new log
I20260927 23:03:51.396981 5085 ts_tablet_manager.cc:1397] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260927 23:03:51.398087 5083 raft_consensus.cc:348] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.398435 5083 raft_consensus.cc:374] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:51.398538 5083 raft_consensus.cc:729] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c09dc8a169463d809dc88fd3fea42c, State: Initialized, Role: FOLLOWER
I20260927 23:03:51.399080 5083 consensus_queue.cc:260] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.400079 5065 heartbeater.cc:503] Master 127.3.240.254:39491 was elected leader, sending a full tablet report...
I20260927 23:03:51.400456 5083 ts_tablet_manager.cc:1428] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260927 23:03:51.406854 5085 raft_consensus.cc:348] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.407295 5085 raft_consensus.cc:374] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:51.407398 5085 raft_consensus.cc:729] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f53119a191e42fbb2c6b8d71c523ddc, State: Initialized, Role: FOLLOWER
I20260927 23:03:51.407544 5084 raft_consensus.cc:348] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.407900 5084 raft_consensus.cc:374] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:51.408010 5084 raft_consensus.cc:729] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05af018fe6184300a8f4dce841bf4e20, State: Initialized, Role: FOLLOWER
I20260927 23:03:51.407982 5085 consensus_queue.cc:260] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.408602 5084 consensus_queue.cc:260] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.409004 4932 heartbeater.cc:503] Master 127.3.240.254:39491 was elected leader, sending a full tablet report...
I20260927 23:03:51.409704 5085 ts_tablet_manager.cc:1428] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260927 23:03:51.410213 5084 ts_tablet_manager.cc:1428] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260927 23:03:51.429454 5089 raft_consensus.cc:482] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:03:51.429704 5089 raft_consensus.cc:504] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.431180 5089 leader_election.cc:288] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f53119a191e42fbb2c6b8d71c523ddc (127.3.240.194:42941), 05af018fe6184300a8f4dce841bf4e20 (127.3.240.193:34919)
I20260927 23:03:51.437389 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.431561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:49113 (local address 127.3.240.194:42941)
0927 23:03:51.431753 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.431759 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.431780 (+ 21us) server_negotiation.cc:407] Connection header received
0927 23:03:51.431929 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.431937 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.432028 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.432134 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:51.432721 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.433684 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.435542 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.436633 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.436752 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.436798 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.437125 (+ 327us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.437219 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 23:03:51.437487 5067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.431511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:42941 (local address 127.3.240.195:49113)
0927 23:03:51.431693 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:51.431710 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:51.431796 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:51.432327 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:51.432334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:51.432365 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:51.432544 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.432556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.433799 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.433805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.435408 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.435419 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.436785 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.436794 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.436965 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.436999 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 23:03:51.437106 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 23:03:51.437226 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 23:03:51.438514 5092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.431920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34919 (local address 127.3.240.195:54379)
0927 23:03:51.432530 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:51.432546 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:51.432656 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:51.432950 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:51.432955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:51.432999 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:51.433319 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.433329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.434644 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.434647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.436371 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:51.436384 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.437871 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.437877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:51.438045 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.438080 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 23:03:51.438209 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 23:03:51.438299 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":273,"threads_started":1}
I20260927 23:03:51.438548 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.432034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:54379 (local address 127.3.240.193:34919)
0927 23:03:51.432257 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.432262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.432619 (+ 357us) server_negotiation.cc:407] Connection header received
0927 23:03:51.432775 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.432779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.432843 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.432913 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:51.433440 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.434506 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.436509 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.437645 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.437813 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.437859 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.438205 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.438324 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 23:03:51.438534 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8454dbde82a344d08907758d422124cc" candidate_uuid: "d8c09dc8a169463d809dc88fd3fea42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" is_pre_election: true
I20260927 23:03:51.439100 4887 raft_consensus.cc:2454] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c09dc8a169463d809dc88fd3fea42c in term 0.
I20260927 23:03:51.439471 4753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8454dbde82a344d08907758d422124cc" candidate_uuid: "d8c09dc8a169463d809dc88fd3fea42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05af018fe6184300a8f4dce841bf4e20" is_pre_election: true
I20260927 23:03:51.439668 4953 leader_election.cc:302] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f53119a191e42fbb2c6b8d71c523ddc, d8c09dc8a169463d809dc88fd3fea42c; no voters:
I20260927 23:03:51.439931 4753 raft_consensus.cc:2454] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c09dc8a169463d809dc88fd3fea42c in term 0.
I20260927 23:03:51.440011 5089 raft_consensus.cc:2781] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:03:51.440135 5089 raft_consensus.cc:482] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:03:51.440248 5089 raft_consensus.cc:3037] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:51.442299 5089 raft_consensus.cc:504] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.442824 5089 leader_election.cc:288] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [CANDIDATE]: Term 1 election: Requested vote from peers 4f53119a191e42fbb2c6b8d71c523ddc (127.3.240.194:42941), 05af018fe6184300a8f4dce841bf4e20 (127.3.240.193:34919)
I20260927 23:03:51.443229 4887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8454dbde82a344d08907758d422124cc" candidate_uuid: "d8c09dc8a169463d809dc88fd3fea42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f53119a191e42fbb2c6b8d71c523ddc"
I20260927 23:03:51.443301 4753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8454dbde82a344d08907758d422124cc" candidate_uuid: "d8c09dc8a169463d809dc88fd3fea42c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05af018fe6184300a8f4dce841bf4e20"
I20260927 23:03:51.443390 4887 raft_consensus.cc:3037] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:51.443514 4753 raft_consensus.cc:3037] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:51.445585 4753 raft_consensus.cc:2454] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c09dc8a169463d809dc88fd3fea42c in term 1.
I20260927 23:03:51.445581 4887 raft_consensus.cc:2454] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c09dc8a169463d809dc88fd3fea42c in term 1.
I20260927 23:03:51.445914 4953 leader_election.cc:302] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05af018fe6184300a8f4dce841bf4e20, d8c09dc8a169463d809dc88fd3fea42c; no voters:
I20260927 23:03:51.446174 5089 raft_consensus.cc:2781] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:51.446568 5089 raft_consensus.cc:686] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [term 1 LEADER]: Becoming Leader. State: Replica: d8c09dc8a169463d809dc88fd3fea42c, State: Running, Role: LEADER
I20260927 23:03:51.446965 5089 consensus_queue.cc:237] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } }
I20260927 23:03:51.451026 4608 catalog_manager.cc:5495] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c reported cstate change: term changed from 0 to 1, leader changed from <none> to d8c09dc8a169463d809dc88fd3fea42c (127.3.240.195). New cstate: current_term: 1 leader_uuid: "d8c09dc8a169463d809dc88fd3fea42c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c09dc8a169463d809dc88fd3fea42c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f53119a191e42fbb2c6b8d71c523ddc" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 42941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05af018fe6184300a8f4dce841bf4e20" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34919 } health_report { overall_health: UNKNOWN } } }
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:03:51.485028 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.471793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41348 (local address 127.3.240.193:34919)
0927 23:03:51.471976 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.472000 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.472397 (+ 397us) server_negotiation.cc:407] Connection header received
0927 23:03:51.472532 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.472535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.472602 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.472712 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:51.474075 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.474796 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.475945 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.476359 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.476465 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.479954 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:51.479965 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:51.479977 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:51.480019 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:51.482558 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.483138 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.483144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.483146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.483251 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.483548 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.483551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.483552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.484232 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:51.484415 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.484573 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.484693 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550231, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
W20260927 23:03:51.523870 5066 tablet.cc:2369] T 8454dbde82a344d08907758d422124cc P d8c09dc8a169463d809dc88fd3fea42c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:03:51.525696 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.512815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41352 (local address 127.3.240.193:34919)
0927 23:03:51.513021 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.513029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.513283 (+ 254us) server_negotiation.cc:407] Connection header received
0927 23:03:51.513470 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.513474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.513536 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.513627 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:51.514818 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.515584 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.516801 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.517196 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.517323 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.520827 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:51.520832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:51.520835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:51.520878 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:51.523300 (+ 2422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.523861 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.523871 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.523873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.523984 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.524417 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.524420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.524422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.524652 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:51.524855 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.525122 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.525384 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
W20260927 23:03:51.562247 4933 tablet.cc:2369] T 8454dbde82a344d08907758d422124cc P 4f53119a191e42fbb2c6b8d71c523ddc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:03:51.565119 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.552085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41368 (local address 127.3.240.193:34919)
0927 23:03:51.552248 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.552255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.552582 (+ 327us) server_negotiation.cc:407] Connection header received
0927 23:03:51.552762 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.552766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.552819 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.552909 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:51.554090 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.554694 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.555946 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.556345 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.556469 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.560019 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:51.560025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:51.560028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:51.560062 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:51.562419 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.563059 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.563068 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.563071 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.563181 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.563546 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.563553 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.563557 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.563896 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:51.564157 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.564508 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.564721 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260927 23:03:51.566956 4713 tablet_service.cc:2908] Rejecting scan request for tablet 8454dbde82a344d08907758d422124cc: Uninitialized: clean time has not yet been initialized
I20260927 23:03:51.569823 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 4670
I20260927 23:03:51.581077 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.570714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.3.240.193:34919)
0927 23:03:51.570892 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.570901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.571230 (+ 329us) server_negotiation.cc:407] Connection header received
0927 23:03:51.571326 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.571330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.571390 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.571463 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:51.572430 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.573073 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.574148 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.574535 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.574630 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.576536 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:51.576540 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:51.576542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:51.576567 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:51.579270 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.579782 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.579785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.579788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.579883 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.580140 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.580143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.580145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.580331 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:51.580517 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.580665 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.580817 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 23:03:51.581657 4793 generic_service.cc:196] Checking for leaks (request via RPC)
W20260927 23:03:51.666527 4799 tablet.cc:2369] T 8454dbde82a344d08907758d422124cc P 05af018fe6184300a8f4dce841bf4e20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:03:51.667235 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 4670
I20260927 23:03:51.685642 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 4804
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:03:51.699744 5080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.686506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.3.240.194:42941)
0927 23:03:51.686680 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.686685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.686703 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:03:51.686762 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.686766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.686830 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.686958 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:51.688125 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.688896 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.690060 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.690422 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.690518 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.694204 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:51.694217 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:51.694232 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:51.694261 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:51.696968 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.697622 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.697629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.697631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.697748 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.698121 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.698126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.698129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.698504 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:51.698821 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.699286 (+ 465us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.699488 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:03:51.700407 4927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:03:51.785768 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 4804
I20260927 23:03:51.804774 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 4937
Sep 27 23:03:51 dist-test-slave-lmfp krb5kdc[4564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:03:51.821201 5079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:51.805549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.3.240.195:39201)
0927 23:03:51.805738 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:51.805742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:51.805763 (+ 21us) server_negotiation.cc:407] Connection header received
0927 23:03:51.805960 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:51.805965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:51.806031 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:51.806121 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:51.807704 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.808524 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.810304 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:51.810691 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:51.810787 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:51.815397 (+ 4610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:51.815415 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:51.815436 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:51.815495 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:51.818906 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.819616 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.819623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.819626 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.819749 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:51.820069 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:51.820072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:51.820074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:51.820392 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:51.820606 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:51.820805 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 23:03:51.820963 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 23:03:51.821805 5060 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:03:51.907147 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 4937
I20260927 23:03:51.923602 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 4577
I20260927 23:03:51.924690 4639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:03:52.027807 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 4577
2026-09-27T23:03:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3833 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:03:52 dist-test-slave-lmfp krb5kdc[5112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:03:52 dist-test-slave-lmfp krb5kdc[5112](info): set up 2 sockets
Sep 27 23:03:52 dist-test-slave-lmfp krb5kdc[5112](info): commencing operation
krb5kdc: starting...
W20260927 23:03:53.066762 4035 mini_kdc.cc:121] Time spent starting KDC: real 0.997s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:03:53 dist-test-slave-lmfp krb5kdc[5112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:03:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:03:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:03:53.241756 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:34511
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:53.542290 5125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:53.542575 5125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:53.542713 5125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:53.549348 5125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:03:53.549463 5125 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 23:03:53.549546 5125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:53.549659 5125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:53.549731 5125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:03:53.549814 5125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:03:53.559211 5125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:34511
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:53.560338 5125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:53.561544 5125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:53.572954 5130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:53.573081 5133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:53.573056 5131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:53.573496 5125 server_base.cc:1034] running on GCE node
I20260927 23:03:53.574388 5125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:53.575680 5125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:53.576906 5125 hybrid_clock.cc:648] HybridClock initialized: now 1790550233576842 us; error 48 us; skew 500 ppm
I20260927 23:03:53.577186 5125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:53 dist-test-slave-lmfp krb5kdc[5112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550233, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:53.580149 5125 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:03:53.581430 5125 webserver.cc:460] Webserver started at http://127.3.240.254:46525/ using document root <none> and password file <none>
I20260927 23:03:53.582006 5125 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:53.582082 5125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:53.582343 5125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:53.584950 5125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "a64c0119b1624364b6337f74500a6e7c"
format_stamp: "Formatted at 2026-09-27 23:03:53 on dist-test-slave-lmfp"
server_key: "334e6b89ba7d3c8474012eb555d0c5fc"
server_key_iv: "68133d96f869f7540f69e27c07b8b8fb"
server_key_version: "encryptionkey@0"
I20260927 23:03:53.585682 5125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "a64c0119b1624364b6337f74500a6e7c"
format_stamp: "Formatted at 2026-09-27 23:03:53 on dist-test-slave-lmfp"
server_key: "334e6b89ba7d3c8474012eb555d0c5fc"
server_key_iv: "68133d96f869f7540f69e27c07b8b8fb"
server_key_version: "encryptionkey@0"
I20260927 23:03:53.590231 5125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 23:03:53.593232 5140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:53.594641 5125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 23:03:53.594781 5125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "a64c0119b1624364b6337f74500a6e7c"
format_stamp: "Formatted at 2026-09-27 23:03:53 on dist-test-slave-lmfp"
server_key: "334e6b89ba7d3c8474012eb555d0c5fc"
server_key_iv: "68133d96f869f7540f69e27c07b8b8fb"
server_key_version: "encryptionkey@0"
I20260927 23:03:53.594959 5125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:53.608245 5125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:53.612151 5125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:53.612399 5125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:53.632166 5125 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:34511
I20260927 23:03:53.632172 5201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:34511 every 8 connection(s)
I20260927 23:03:53.633468 5125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:03:53.635684 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 5125
I20260927 23:03:53.635900 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:03:53.636523 4035 external_mini_cluster.cc:1428] Setting key 196441a3905716ae5e2b049f7ffaefd6
I20260927 23:03:53.637146 5202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:53.650074 5202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c: Bootstrap starting.
Sep 27 23:03:53 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:53.653414 5202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:53.654618 5202 log.cc:824] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:53.657639 5202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c: No bootstrap required, opened a new log
I20260927 23:03:53.659204 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:53.638419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.3.240.254:34511)
0927 23:03:53.639214 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:53.639229 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:53.639274 (+ 45us) server_negotiation.cc:407] Connection header received
0927 23:03:53.641583 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:53.641610 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:53.642101 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:53.642535 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:53.643794 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:53.645668 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:53.646851 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:53.647880 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:53.648067 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:53.651463 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:53.651473 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:53.651498 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:53.651547 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:53.655135 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:53.655908 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:53.655917 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:53.655930 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:53.656100 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:53.656393 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:53.656397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:53.656399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:53.656964 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:53.657237 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:53.657538 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 23:03:53.657942 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":348,"threads_started":1}
I20260927 23:03:53.665874 5202 raft_consensus.cc:348] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64c0119b1624364b6337f74500a6e7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34511 } }
I20260927 23:03:53.666172 5202 raft_consensus.cc:374] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:53.666239 5202 raft_consensus.cc:729] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a64c0119b1624364b6337f74500a6e7c, State: Initialized, Role: FOLLOWER
I20260927 23:03:53.666743 5202 consensus_queue.cc:260] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64c0119b1624364b6337f74500a6e7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34511 } }
I20260927 23:03:53.666945 5202 raft_consensus.cc:388] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:03:53.667088 5202 raft_consensus.cc:482] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:03:53.667235 5202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:53.669353 5202 raft_consensus.cc:504] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64c0119b1624364b6337f74500a6e7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34511 } }
I20260927 23:03:53.669806 5202 leader_election.cc:302] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a64c0119b1624364b6337f74500a6e7c; no voters:
I20260927 23:03:53.670368 5202 leader_election.cc:288] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:03:53.670526 5207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:53.671356 5207 raft_consensus.cc:686] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [term 1 LEADER]: Becoming Leader. State: Replica: a64c0119b1624364b6337f74500a6e7c, State: Running, Role: LEADER
I20260927 23:03:53.671819 5207 consensus_queue.cc:237] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64c0119b1624364b6337f74500a6e7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34511 } }
I20260927 23:03:53.672313 5202 sys_catalog.cc:564] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:03:53.674803 5209 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a64c0119b1624364b6337f74500a6e7c. Latest consensus state: current_term: 1 leader_uuid: "a64c0119b1624364b6337f74500a6e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64c0119b1624364b6337f74500a6e7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34511 } } }
I20260927 23:03:53.675251 5209 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [sys.catalog]: This master's current role is: LEADER
I20260927 23:03:53.678423 5208 sys_catalog.cc:455] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a64c0119b1624364b6337f74500a6e7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a64c0119b1624364b6337f74500a6e7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34511 } } }
I20260927 23:03:53.678669 5208 sys_catalog.cc:458] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c [sys.catalog]: This master's current role is: LEADER
I20260927 23:03:53.679237 5216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:03:53.684103 5216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:03:53.692190 5216 catalog_manager.cc:1347] Generated new cluster ID: 6215a5e8d6d14cefa19c306f94c40409
I20260927 23:03:53.692291 5216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:03:53.724036 5216 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:03:53.725302 5216 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:03:53.732653 5216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c: Generated new TSK 0
I20260927 23:03:53.733505 5216 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:03:53.786288 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34511
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:54.100360 5228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:54.100616 5228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:54.100769 5228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:54.107726 5228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:54.107817 5228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:54.108001 5228 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:03:54.108132 5228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:03:54.117424 5228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:34511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:54.118665 5228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:54.120041 5228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:54.132548 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:54.132619 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:54.132589 5236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:54.133477 5228 server_base.cc:1034] running on GCE node
I20260927 23:03:54.134088 5228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:54.134966 5228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:54.136233 5228 hybrid_clock.cc:648] HybridClock initialized: now 1790550234136173 us; error 36 us; skew 500 ppm
I20260927 23:03:54.136514 5228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:54 dist-test-slave-lmfp krb5kdc[5112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550234, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:54.139524 5228 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:03:54.140798 5228 webserver.cc:460] Webserver started at http://127.3.240.193:39847/ using document root <none> and password file <none>
I20260927 23:03:54.141446 5228 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:54.141567 5228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:54.141855 5228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:54.144644 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf"
format_stamp: "Formatted at 2026-09-27 23:03:54 on dist-test-slave-lmfp"
server_key: "8639c09b54a6e440f4c5c7519506e791"
server_key_iv: "9347d4fa54efd40e2a3692b1ef9c960e"
server_key_version: "encryptionkey@0"
I20260927 23:03:54.145381 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf"
format_stamp: "Formatted at 2026-09-27 23:03:54 on dist-test-slave-lmfp"
server_key: "8639c09b54a6e440f4c5c7519506e791"
server_key_iv: "9347d4fa54efd40e2a3692b1ef9c960e"
server_key_version: "encryptionkey@0"
I20260927 23:03:54.150110 5228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 23:03:54.153533 5243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:54.154907 5228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:03:54.155108 5228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf"
format_stamp: "Formatted at 2026-09-27 23:03:54 on dist-test-slave-lmfp"
server_key: "8639c09b54a6e440f4c5c7519506e791"
server_key_iv: "9347d4fa54efd40e2a3692b1ef9c960e"
server_key_version: "encryptionkey@0"
I20260927 23:03:54.155356 5228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:54.179833 5228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:54.183969 5228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:54.184250 5228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:54.186746 5228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:54.186859 5228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:54.187052 5228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:54.187126 5228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 27 23:03:54 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550234, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:54.210100 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:54.188008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:37361 (local address 127.3.240.254:34511)
0927 23:03:54.188494 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:54.188502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:54.189308 (+ 806us) server_negotiation.cc:407] Connection header received
0927 23:03:54.191362 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:54.191371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:54.191470 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:54.191607 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:54.193549 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.194618 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.197125 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.197588 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.197726 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:54.202391 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:54.202396 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:54.202399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:54.202428 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:54.205752 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:54.206746 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:54.206754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:54.206761 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:54.206943 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:54.207649 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:54.207654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:54.207658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:54.208047 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:54.208190 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:54.209264 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0927 23:03:54.209568 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":251,"threads_started":1}
I20260927 23:03:54.210602 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:54.188210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34511 (local address 127.3.240.193:37361)
0927 23:03:54.189132 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:54.189178 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:54.191083 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:54.191837 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:54.191844 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:54.192402 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:54.193343 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:54.193389 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.194762 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.194769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:54.196981 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:54.196995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.197840 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.197850 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:54.198078 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:54.198924 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:54.198961 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:54.202186 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:54.205948 (+ 3762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:54.205961 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:54.205987 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:54.206572 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:54.207115 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:54.207123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:54.207129 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:54.207509 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:54.208160 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:54.208171 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:54.208724 (+ 553us) client_negotiation.cc:769] Sending connection context
0927 23:03:54.209111 (+ 387us) client_negotiation.cc:240] Negotiation successful
0927 23:03:54.209506 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":394,"threads_started":1}
I20260927 23:03:54.231226 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:54.217238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57259 (local address 127.3.240.254:34511)
0927 23:03:54.217463 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:54.217471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:54.217981 (+ 510us) server_negotiation.cc:407] Connection header received
0927 23:03:54.218118 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:54.218125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:54.218219 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:54.218348 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:54.219235 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.220241 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.222293 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.222936 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.223110 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:54.225850 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:54.225855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:54.225858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:54.225885 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:54.228385 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:54.229310 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:54.229313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:54.229315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:54.229434 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:54.229842 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:54.229849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:54.229852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:54.230144 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:54.230359 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:54.230802 (+ 443us) server_negotiation.cc:299] Negotiation successful
0927 23:03:54.230950 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:03:54.231652 5322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:54.217233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34511 (local address 127.3.240.193:57259)
0927 23:03:54.217888 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:54.217909 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:54.218013 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:54.218354 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:54.218357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:54.218768 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:54.219090 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:54.219098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.220363 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.220370 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:54.222162 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:54.222174 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.223178 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.223190 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:54.223454 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:54.224287 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:54.224318 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:54.225641 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:54.228511 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:54.228518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:54.228533 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:54.229189 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:54.229537 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:54.229542 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:54.229547 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:54.229724 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:54.230344 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:54.230350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:54.230568 (+ 218us) client_negotiation.cc:769] Sending connection context
0927 23:03:54.230715 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 23:03:54.231072 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":346,"threads_started":1}
I20260927 23:03:54.242220 5228 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:44525
I20260927 23:03:54.242242 5363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:44525 every 8 connection(s)
I20260927 23:03:54.243446 5228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:03:54.244004 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 5228
I20260927 23:03:54.244225 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:03:54.245026 4035 external_mini_cluster.cc:1428] Setting key ac13eab17e8cce6adeefed7bbf2ccdbb
I20260927 23:03:54.245464 5364 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34511
I20260927 23:03:54.245687 5364 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:54.246284 5364 heartbeater.cc:511] Master 127.3.240.254:34511 requested a full tablet report, sending...
I20260927 23:03:54.248452 5157 ts_manager.cc:194] Registered new tserver with Master: f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525)
I20260927 23:03:54.250552 5157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:37361
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:03:54.281617 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34511
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:54.592828 5368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:54.593150 5368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:54.593314 5368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:54.600517 5368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:54.600654 5368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:54.600790 5368 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:03:54.600948 5368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:03:54.610255 5368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:34511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:54.611838 5368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:54.613695 5368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:54.626552 5374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:54.626503 5376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:54.626569 5373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:54.627334 5368 server_base.cc:1034] running on GCE node
I20260927 23:03:54.627863 5368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:54.628670 5368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:54.629891 5368 hybrid_clock.cc:648] HybridClock initialized: now 1790550234629831 us; error 39 us; skew 500 ppm
I20260927 23:03:54.630178 5368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:54 dist-test-slave-lmfp krb5kdc[5112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550234, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:54.633505 5368 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:03:54.644026 5368 webserver.cc:460] Webserver started at http://127.3.240.194:44755/ using document root <none> and password file <none>
I20260927 23:03:54.644732 5368 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:54.644834 5368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:54.645144 5368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:54.647866 5368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "fc3863d43f0d4427a3240aeda0b57274"
format_stamp: "Formatted at 2026-09-27 23:03:54 on dist-test-slave-lmfp"
server_key: "bf06f47d0b51e7634f2ed152a3d22489"
server_key_iv: "6ddd89b1f72440ca4e88873c14c3b806"
server_key_version: "encryptionkey@0"
I20260927 23:03:54.648608 5368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "fc3863d43f0d4427a3240aeda0b57274"
format_stamp: "Formatted at 2026-09-27 23:03:54 on dist-test-slave-lmfp"
server_key: "bf06f47d0b51e7634f2ed152a3d22489"
server_key_iv: "6ddd89b1f72440ca4e88873c14c3b806"
server_key_version: "encryptionkey@0"
I20260927 23:03:54.653292 5368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:03:54.656531 5383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:54.657677 5368 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:03:54.657824 5368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "fc3863d43f0d4427a3240aeda0b57274"
format_stamp: "Formatted at 2026-09-27 23:03:54 on dist-test-slave-lmfp"
server_key: "bf06f47d0b51e7634f2ed152a3d22489"
server_key_iv: "6ddd89b1f72440ca4e88873c14c3b806"
server_key_version: "encryptionkey@0"
I20260927 23:03:54.658003 5368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:54.671808 5368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:54.675596 5368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:54.675827 5368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:54.678035 5368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:54.678130 5368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:54.678241 5368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:54.678375 5368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 23:03:54 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550234, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:54.699316 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:54.678702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:44969 (local address 127.3.240.254:34511)
0927 23:03:54.679808 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:54.679815 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:54.680198 (+ 383us) server_negotiation.cc:407] Connection header received
0927 23:03:54.682261 (+ 2063us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:54.682266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:54.682342 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:54.682489 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:54.683958 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.685176 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.687411 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.687970 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.688080 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:54.692494 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:54.692498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:54.692501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:54.692529 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:54.695611 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:54.696520 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:54.696525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:54.696529 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:54.696690 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:54.697159 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:54.697165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:54.697170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:54.697476 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:54.697581 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:54.698725 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0927 23:03:54.698923 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974}
I20260927 23:03:54.700778 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:54.679313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34511 (local address 127.3.240.194:44969)
0927 23:03:54.680058 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:54.680096 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:54.681953 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:54.682771 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:54.682779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:54.683271 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:54.683794 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:54.683810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.685329 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.685332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:54.687281 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:54.687291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.688357 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.688366 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:54.688652 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:54.689395 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:54.689417 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:54.692244 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:54.695734 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:54.695742 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:54.695773 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:54.696375 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:54.696824 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:54.696831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:54.696834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:54.697009 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:54.697725 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:54.697770 (+ 45us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:54.698290 (+ 520us) client_negotiation.cc:769] Sending connection context
0927 23:03:54.698629 (+ 339us) client_negotiation.cc:240] Negotiation successful
0927 23:03:54.699228 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":400,"threads_started":1}
I20260927 23:03:54.718974 5368 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:39957
I20260927 23:03:54.719060 5502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:39957 every 8 connection(s)
I20260927 23:03:54.720199 5368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:03:54.722385 5503 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34511
I20260927 23:03:54.722638 5503 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:54.723495 5503 heartbeater.cc:511] Master 127.3.240.254:34511 requested a full tablet report, sending...
I20260927 23:03:54.724802 5157 ts_manager.cc:194] Registered new tserver with Master: fc3863d43f0d4427a3240aeda0b57274 (127.3.240.194:39957)
I20260927 23:03:54.725787 5157 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:44969
I20260927 23:03:54.726262 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 5368
I20260927 23:03:54.726456 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:03:54.726969 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:54.714602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42631 (local address 127.3.240.254:34511)
0927 23:03:54.714769 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:54.714775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:54.714958 (+ 183us) server_negotiation.cc:407] Connection header received
0927 23:03:54.715201 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:54.715208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:54.715300 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:54.715395 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:54.716693 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.717536 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.719143 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.719538 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.719663 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:54.721667 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:54.721673 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:54.721677 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:54.721711 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:54.724555 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:54.725191 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:54.725197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:54.725204 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:54.725338 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:54.725648 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:54.725652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:54.725654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:54.726051 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:54.726172 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:54.726518 (+ 346us) server_negotiation.cc:299] Negotiation successful
0927 23:03:54.726693 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 23:03:54.726980 4035 external_mini_cluster.cc:1428] Setting key 952cde57217bcd496504fb7889f80ea3
I20260927 23:03:54.727150 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:54.714085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34511 (local address 127.3.240.194:42631)
0927 23:03:54.714829 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:54.714863 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:54.715060 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:54.715413 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:54.715417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:54.716088 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:54.716507 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:54.716525 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.717662 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.717667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:54.718937 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:54.718947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:54.719703 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:54.719708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:54.719874 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:54.720421 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:54.720435 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:54.721457 (+ 1022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:54.724673 (+ 3216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:54.724676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:54.724679 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:54.725085 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:54.725440 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:54.725443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:54.725445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:54.725564 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:54.726144 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:54.726148 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:54.726309 (+ 161us) client_negotiation.cc:769] Sending connection context
0927 23:03:54.726447 (+ 138us) client_negotiation.cc:240] Negotiation successful
0927 23:03:54.726706 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":553,"threads_started":1}
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:03:54.764300 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34511
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:55.060585 5507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:55.060849 5507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:55.061018 5507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:55.068001 5507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:55.068127 5507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:55.068275 5507 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:03:55.068397 5507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:03:55.077803 5507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:34511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:55.079011 5507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:55.080331 5507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:55.092514 5512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:55.092543 5513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:55.092890 5515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:55.093546 5507 server_base.cc:1034] running on GCE node
I20260927 23:03:55.094149 5507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:55.094960 5507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:55.096220 5507 hybrid_clock.cc:648] HybridClock initialized: now 1790550235096165 us; error 47 us; skew 500 ppm
I20260927 23:03:55.096485 5507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:55 dist-test-slave-lmfp krb5kdc[5112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550235, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:55.099380 5507 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:03:55.100695 5507 webserver.cc:460] Webserver started at http://127.3.240.195:42489/ using document root <none> and password file <none>
I20260927 23:03:55.101346 5507 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:55.101456 5507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:55.101738 5507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:55.105196 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "a4ed31ff721d4f47827cb8a5647e1def"
format_stamp: "Formatted at 2026-09-27 23:03:55 on dist-test-slave-lmfp"
server_key: "d2031472c5f3df22d98ed572c4766d2f"
server_key_iv: "5a28712523c651113c31fd7e0b0d5f0f"
server_key_version: "encryptionkey@0"
I20260927 23:03:55.106062 5507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "a4ed31ff721d4f47827cb8a5647e1def"
format_stamp: "Formatted at 2026-09-27 23:03:55 on dist-test-slave-lmfp"
server_key: "d2031472c5f3df22d98ed572c4766d2f"
server_key_iv: "5a28712523c651113c31fd7e0b0d5f0f"
server_key_version: "encryptionkey@0"
I20260927 23:03:55.111961 5507 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260927 23:03:55.115060 5522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:55.116148 5507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:03:55.116289 5507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "a4ed31ff721d4f47827cb8a5647e1def"
format_stamp: "Formatted at 2026-09-27 23:03:55 on dist-test-slave-lmfp"
server_key: "d2031472c5f3df22d98ed572c4766d2f"
server_key_iv: "5a28712523c651113c31fd7e0b0d5f0f"
server_key_version: "encryptionkey@0"
I20260927 23:03:55.116490 5507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:55.129748 5507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:55.133710 5507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:55.133955 5507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:55.136312 5507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:55.136401 5507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:55.136569 5507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:55.136626 5507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 23:03:55 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550235, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:55.161091 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.137033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:53849 (local address 127.3.240.254:34511)
0927 23:03:55.137225 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.137231 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.138574 (+ 1343us) server_negotiation.cc:407] Connection header received
0927 23:03:55.141167 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.141175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.141297 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.141414 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:55.142979 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.144286 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.146660 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.147279 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.147435 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.152820 (+ 5385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:55.152824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:55.152828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:55.152857 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:55.155967 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.156872 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.156879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.156885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.157087 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.157774 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.157781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.157786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.158232 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:55.158401 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.160508 (+ 2107us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.160738 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 23:03:55.161540 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.137556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34511 (local address 127.3.240.195:53849)
0927 23:03:55.138381 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.138429 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.140764 (+ 2335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.141666 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.141673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.142132 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:55.142766 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.142791 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.144450 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.144459 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.146494 (+ 2035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.146505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.147532 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.147539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.147865 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.148888 (+ 1023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:55.148924 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:55.152546 (+ 3622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:55.156102 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:55.156117 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:55.156137 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:55.156742 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:55.157241 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:55.157254 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:55.157260 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:55.157583 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:55.158407 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:55.158417 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:55.158918 (+ 501us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.159369 (+ 451us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.159964 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":362,"threads_started":1}
I20260927 23:03:55.177524 5507 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42147
I20260927 23:03:55.177557 5641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42147 every 8 connection(s)
I20260927 23:03:55.178614 5507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:03:55.180863 5642 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34511
I20260927 23:03:55.181069 5642 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:55.181690 5642 heartbeater.cc:511] Master 127.3.240.254:34511 requested a full tablet report, sending...
I20260927 23:03:55.183039 5156 ts_manager.cc:194] Registered new tserver with Master: a4ed31ff721d4f47827cb8a5647e1def (127.3.240.195:42147)
I20260927 23:03:55.183974 5156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:53849
I20260927 23:03:55.188596 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 5507
I20260927 23:03:55.188784 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:03:55.189193 4035 external_mini_cluster.cc:1428] Setting key f8293e58efd9f508f3a4ff58ee5c4705
I20260927 23:03:55.191275 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.169736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47847 (local address 127.3.240.254:34511)
0927 23:03:55.170284 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.170292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.180138 (+ 9846us) server_negotiation.cc:407] Connection header received
0927 23:03:55.180255 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.180260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.180344 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.180415 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:55.181428 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.182195 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.183787 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.184287 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.184404 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.186144 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:55.186150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:55.186153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:55.186188 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:55.188611 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.189317 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.189334 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.189341 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.189460 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.189834 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.189837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.189839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.190141 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:55.190302 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.190691 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.190888 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398}
I20260927 23:03:55.191349 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.169793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34511 (local address 127.3.240.195:47847)
0927 23:03:55.180012 (+ 10219us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.180030 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.180165 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.180478 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.180482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.181003 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:55.181300 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.181310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.182318 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.182325 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.183670 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.183681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.184476 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.184483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.184684 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.185286 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:55.185303 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:55.185973 (+ 670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:55.188754 (+ 2781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:55.188759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:55.188763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:55.189177 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:55.189561 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:55.189567 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:55.189571 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:55.189722 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:55.190271 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:55.190278 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:55.190454 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.190597 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.190866 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10089,"thread_start_us":9823,"threads_started":1}
I20260927 23:03:55.192157 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 23:03:55 dist-test-slave-lmfp krb5kdc[5112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 23:03:55 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:55.232966 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.220858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.3.240.254:34511)
0927 23:03:55.221094 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.221100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.221325 (+ 225us) server_negotiation.cc:407] Connection header received
0927 23:03:55.221465 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.221482 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.221577 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.221695 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:55.222877 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.223645 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.224773 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.225147 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.225257 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.228562 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:55.228566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:55.228569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:55.228593 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:55.230903 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.231510 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.231513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.231515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.231626 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.231891 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.231894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.231895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.232109 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:55.232264 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.232437 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.232581 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 23:03:55.236418 5156 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:03:55.242868 5156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:03:55.253265 5364 heartbeater.cc:503] Master 127.3.240.254:34511 was elected leader, sending a full tablet report...
I20260927 23:03:55.274053 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.253704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57884 (local address 127.3.240.194:39957)
0927 23:03:55.266533 (+ 12829us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.266542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.266569 (+ 27us) server_negotiation.cc:407] Connection header received
0927 23:03:55.266664 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.266672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.266909 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.267171 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:55.267728 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.269245 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.271690 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.272980 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.273420 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.273598 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.273668 (+ 70us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.273826 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12651,"thread_start_us":12538,"threads_started":1}
I20260927 23:03:55.274103 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.253525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:39957 (local address 127.0.0.1:57884)
0927 23:03:55.264428 (+ 10903us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.264477 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.265549 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.267171 (+ 1622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.267178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.267204 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:55.267525 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.267539 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.269353 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.269361 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.271519 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.271532 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.273210 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.273216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.273431 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.273475 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.273560 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.273682 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10548,"mutex_wait_us":25,"thread_start_us":13091,"threads_started":1}
I20260927 23:03:55.274938 5655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.253781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44525 (local address 127.0.0.1:35228)
0927 23:03:55.266951 (+ 13170us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.266969 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.267109 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.267921 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.267925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.267947 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:55.268478 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.268489 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.270129 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.270134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.271756 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.271766 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.273111 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.273116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.274354 (+ 1238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.274386 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.274463 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.274578 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13006,"mutex_wait_us":5,"thread_start_us":10831,"threads_started":1}
I20260927 23:03:55.275691 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.254015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.3.240.193:44525)
0927 23:03:55.266107 (+ 12092us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.266116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.267063 (+ 947us) server_negotiation.cc:407] Connection header received
0927 23:03:55.267286 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.267293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.267470 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.267666 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:55.268624 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.270021 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.271880 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.272916 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.273075 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.273278 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.275270 (+ 1992us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.275415 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11969,"thread_start_us":11853,"threads_started":1}
I20260927 23:03:55.276568 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.267330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.3.240.195:42147)
0927 23:03:55.267931 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.267938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.268619 (+ 681us) server_negotiation.cc:407] Connection header received
0927 23:03:55.268862 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.268870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.269076 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.269240 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:55.270098 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.271566 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.273883 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.275366 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.275476 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.275684 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.276120 (+ 436us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.276261 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":297,"threads_started":1}
I20260927 23:03:55.276614 5660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.267895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42147 (local address 127.0.0.1:33728)
0927 23:03:55.268526 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.268543 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.268701 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.269498 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.269502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.269527 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:55.269877 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.269886 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.271702 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.271708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.273727 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.273739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.275585 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.275591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.275918 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.275983 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.276116 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.276224 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":17,"thread_start_us":336,"threads_started":1}
I20260927 23:03:55.277895 5433 tablet_service.cc:1467] Processing CreateTablet for tablet 65d2ca393d3a41a88b0fc268430d2fba (DEFAULT_TABLE table=test-table [id=3ded869cb0324c1595932bfb6f889f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:55.279057 5433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d2ca393d3a41a88b0fc268430d2fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:55.279678 5572 tablet_service.cc:1467] Processing CreateTablet for tablet 65d2ca393d3a41a88b0fc268430d2fba (DEFAULT_TABLE table=test-table [id=3ded869cb0324c1595932bfb6f889f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:55.279606 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 65d2ca393d3a41a88b0fc268430d2fba (DEFAULT_TABLE table=test-table [id=3ded869cb0324c1595932bfb6f889f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:55.280827 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d2ca393d3a41a88b0fc268430d2fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:55.280826 5294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65d2ca393d3a41a88b0fc268430d2fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:55.289218 5661 tablet_bootstrap.cc:492] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: Bootstrap starting.
I20260927 23:03:55.292366 5661 tablet_bootstrap.cc:654] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.293717 5662 tablet_bootstrap.cc:492] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Bootstrap starting.
I20260927 23:03:55.293951 5661 log.cc:824] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:55.295269 5663 tablet_bootstrap.cc:492] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: Bootstrap starting.
I20260927 23:03:55.297355 5662 tablet_bootstrap.cc:654] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.297554 5663 tablet_bootstrap.cc:654] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.298820 5662 log.cc:824] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:55.298929 5663 log.cc:824] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:55.301366 5663 tablet_bootstrap.cc:492] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: No bootstrap required, opened a new log
I20260927 23:03:55.301468 5662 tablet_bootstrap.cc:492] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: No bootstrap required, opened a new log
I20260927 23:03:55.301470 5661 tablet_bootstrap.cc:492] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: No bootstrap required, opened a new log
I20260927 23:03:55.301630 5663 ts_tablet_manager.cc:1397] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:03:55.301714 5662 ts_tablet_manager.cc:1397] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260927 23:03:55.301715 5661 ts_tablet_manager.cc:1397] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260927 23:03:55.308512 5663 raft_consensus.cc:348] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.308840 5663 raft_consensus.cc:374] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.308954 5663 raft_consensus.cc:729] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ed31ff721d4f47827cb8a5647e1def, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.309834 5663 consensus_queue.cc:260] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.310216 5667 raft_consensus.cc:482] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:03:55.310578 5667 raft_consensus.cc:504] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.312588 5662 raft_consensus.cc:348] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.312587 5661 raft_consensus.cc:348] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.312984 5661 raft_consensus.cc:374] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.312994 5662 raft_consensus.cc:374] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.313081 5662 raft_consensus.cc:729] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c1d1bbb7f54ba7b1ae64fbe953f9bf, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.313079 5661 raft_consensus.cc:729] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3863d43f0d4427a3240aeda0b57274, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.313634 5662 consensus_queue.cc:260] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.313683 5661 consensus_queue.cc:260] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.317996 5663 ts_tablet_manager.cc:1428] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260927 23:03:55.318333 5642 heartbeater.cc:503] Master 127.3.240.254:34511 was elected leader, sending a full tablet report...
I20260927 23:03:55.321275 5662 ts_tablet_manager.cc:1428] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Time spent starting tablet: real 0.019s user 0.016s sys 0.005s
I20260927 23:03:55.321452 5661 ts_tablet_manager.cc:1428] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260927 23:03:55.321817 5503 heartbeater.cc:503] Master 127.3.240.254:34511 was elected leader, sending a full tablet report...
I20260927 23:03:55.324164 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.313299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:39957 (local address 127.3.240.195:37811)
0927 23:03:55.313974 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.313996 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.314131 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.317061 (+ 2930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.317066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.317096 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:55.317303 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.317315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.319059 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.319067 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.320909 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.320925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.323405 (+ 2480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.323415 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.323589 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.323633 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.323739 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.323868 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497}
I20260927 23:03:55.324633 5656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.314554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:37811 (local address 127.3.240.194:39957)
0927 23:03:55.314781 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.314788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.314810 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:03:55.314884 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.314891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.314992 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.315132 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:55.317516 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.318722 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.321924 (+ 3202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.323083 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.323250 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.323319 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.324412 (+ 1093us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.324498 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:03:55.325172 5667 leader_election.cc:288] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc3863d43f0d4427a3240aeda0b57274 (127.3.240.194:39957), f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525)
I20260927 23:03:55.325888 5453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d2ca393d3a41a88b0fc268430d2fba" candidate_uuid: "a4ed31ff721d4f47827cb8a5647e1def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3863d43f0d4427a3240aeda0b57274" is_pre_election: true
I20260927 23:03:55.326860 5453 raft_consensus.cc:2454] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4ed31ff721d4f47827cb8a5647e1def in term 0.
I20260927 23:03:55.327628 5526 leader_election.cc:302] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ed31ff721d4f47827cb8a5647e1def, fc3863d43f0d4427a3240aeda0b57274; no voters:
I20260927 23:03:55.328004 5667 raft_consensus.cc:2781] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:03:55.328140 5667 raft_consensus.cc:482] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:03:55.328233 5667 raft_consensus.cc:3037] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.330171 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.323959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:35803 (local address 127.3.240.193:44525)
0927 23:03:55.324173 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.324180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.324943 (+ 763us) server_negotiation.cc:407] Connection header received
0927 23:03:55.325111 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.325115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.325179 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.325257 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:55.325738 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.326582 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.328227 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.329293 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.329423 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.329485 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.329880 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.329979 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:03:55.330224 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.323476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44525 (local address 127.3.240.195:35803)
0927 23:03:55.324872 (+ 1396us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.324887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.325019 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.325406 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.325410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.325430 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:55.325595 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.325602 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.326707 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.326711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.328089 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.328099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.329593 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.329599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.329743 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.329763 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.329862 (+ 99us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.329968 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1316,"thread_start_us":2602,"threads_started":1}
I20260927 23:03:55.330891 5667 raft_consensus.cc:504] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.331254 5314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d2ca393d3a41a88b0fc268430d2fba" candidate_uuid: "a4ed31ff721d4f47827cb8a5647e1def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" is_pre_election: true
I20260927 23:03:55.331684 5314 raft_consensus.cc:2454] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4ed31ff721d4f47827cb8a5647e1def in term 0.
I20260927 23:03:55.332139 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d2ca393d3a41a88b0fc268430d2fba" candidate_uuid: "a4ed31ff721d4f47827cb8a5647e1def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf"
I20260927 23:03:55.332219 5453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65d2ca393d3a41a88b0fc268430d2fba" candidate_uuid: "a4ed31ff721d4f47827cb8a5647e1def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3863d43f0d4427a3240aeda0b57274"
I20260927 23:03:55.332420 5453 raft_consensus.cc:3037] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.332432 5313 raft_consensus.cc:3037] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.336045 5313 raft_consensus.cc:2454] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4ed31ff721d4f47827cb8a5647e1def in term 1.
I20260927 23:03:55.336043 5453 raft_consensus.cc:2454] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4ed31ff721d4f47827cb8a5647e1def in term 1.
I20260927 23:03:55.336272 5667 leader_election.cc:288] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [CANDIDATE]: Term 1 election: Requested vote from peers fc3863d43f0d4427a3240aeda0b57274 (127.3.240.194:39957), f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525)
I20260927 23:03:55.336467 5526 leader_election.cc:302] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ed31ff721d4f47827cb8a5647e1def, fc3863d43f0d4427a3240aeda0b57274; no voters:
I20260927 23:03:55.336788 5667 raft_consensus.cc:2781] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:55.336988 5667 raft_consensus.cc:686] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 1 LEADER]: Becoming Leader. State: Replica: a4ed31ff721d4f47827cb8a5647e1def, State: Running, Role: LEADER
I20260927 23:03:55.337406 5667 consensus_queue.cc:237] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.341846 5157 catalog_manager.cc:5495] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def reported cstate change: term changed from 0 to 1, leader changed from <none> to a4ed31ff721d4f47827cb8a5647e1def (127.3.240.195). New cstate: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } health_report { overall_health: HEALTHY } } }
I20260927 23:03:55.367689 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.362929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.3.240.195:42147)
0927 23:03:55.363138 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.363142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.363161 (+ 19us) server_negotiation.cc:407] Connection header received
0927 23:03:55.363395 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.363398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.363462 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.363566 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:03:55.363883 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.364558 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.365812 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.366195 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.366281 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.366668 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:03:55.367142 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:03:55.367240 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.367352 (+ 112us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.367419 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 23:03:55.375249 5453 raft_consensus.cc:1261] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Refusing update from remote peer a4ed31ff721d4f47827cb8a5647e1def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:03:55.375267 5313 raft_consensus.cc:1261] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Refusing update from remote peer a4ed31ff721d4f47827cb8a5647e1def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:03:55.376073 5668 consensus_queue.cc:1035] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:03:55.376364 5667 consensus_queue.cc:1035] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:03:55.386196 5674 mvcc.cc:205] Tried to move back new op lower bound from 7334093764086079488 to 7334093763945734144. Current Snapshot: MvccSnapshot[applied={T|T < 7334093764086079488}]
I20260927 23:03:55.386734 5675 mvcc.cc:205] Tried to move back new op lower bound from 7334093764086079488 to 7334093763945734144. Current Snapshot: MvccSnapshot[applied={T|T < 7334093764086079488}]
I20260927 23:03:55.405545 5155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:03:55.405808 5155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:03:55.409210 5155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c: Sending DeleteTablet for 3 replicas of tablet 65d2ca393d3a41a88b0fc268430d2fba
I20260927 23:03:55.410238 5433 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d2ca393d3a41a88b0fc268430d2fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:03:55 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57884
I20260927 23:03:55.410236 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d2ca393d3a41a88b0fc268430d2fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:03:55 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:35228
I20260927 23:03:55.411096 5572 tablet_service.cc:1514] Processing DeleteTablet for tablet 65d2ca393d3a41a88b0fc268430d2fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:03:55 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:33728
I20260927 23:03:55.411631 5684 tablet_replica.cc:330] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: stopping tablet replica
I20260927 23:03:55.412086 5684 raft_consensus.cc:2229] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:03:55.412601 5684 raft_consensus.cc:2258] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:03:55.412959 5155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:03:55.413790 5155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:03:55.422622 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 34fc5b475a324795af6d2f291f49547d (DEFAULT_TABLE table=test-table [id=c5320657d28e4f64a734291365b9d7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:55.422851 5433 tablet_service.cc:1467] Processing CreateTablet for tablet 34fc5b475a324795af6d2f291f49547d (DEFAULT_TABLE table=test-table [id=c5320657d28e4f64a734291365b9d7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:55.422853 5571 tablet_service.cc:1467] Processing CreateTablet for tablet 34fc5b475a324795af6d2f291f49547d (DEFAULT_TABLE table=test-table [id=c5320657d28e4f64a734291365b9d7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:55.423147 5293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fc5b475a324795af6d2f291f49547d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:55.423290 5433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fc5b475a324795af6d2f291f49547d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:55.423519 5571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34fc5b475a324795af6d2f291f49547d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:55.429107 5662 tablet_bootstrap.cc:492] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Bootstrap starting.
I20260927 23:03:55.430951 5662 tablet_bootstrap.cc:654] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.431216 5661 tablet_bootstrap.cc:492] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: Bootstrap starting.
I20260927 23:03:55.431313 5663 tablet_bootstrap.cc:492] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: Bootstrap starting.
I20260927 23:03:55.432968 5663 tablet_bootstrap.cc:654] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.433037 5661 tablet_bootstrap.cc:654] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.437917 5684 ts_tablet_manager.cc:1905] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:03:55.440045 5663 tablet_bootstrap.cc:492] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: No bootstrap required, opened a new log
I20260927 23:03:55.440215 5663 ts_tablet_manager.cc:1397] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260927 23:03:55.441203 5663 raft_consensus.cc:348] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.441466 5663 raft_consensus.cc:374] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.441565 5663 raft_consensus.cc:729] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ed31ff721d4f47827cb8a5647e1def, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.441869 5663 consensus_queue.cc:260] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.442524 5663 ts_tablet_manager.cc:1428] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 23:03:55.444902 5686 tablet_replica.cc:330] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: stopping tablet replica
I20260927 23:03:55.445333 5686 raft_consensus.cc:2229] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:03:55.445652 5661 tablet_bootstrap.cc:492] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: No bootstrap required, opened a new log
I20260927 23:03:55.445787 5661 ts_tablet_manager.cc:1397] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20260927 23:03:55.445829 5686 raft_consensus.cc:2258] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:03:55.446775 5685 tablet_replica.cc:330] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: stopping tablet replica
I20260927 23:03:55.446712 5661 raft_consensus.cc:348] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.446950 5661 raft_consensus.cc:374] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.447080 5661 raft_consensus.cc:729] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3863d43f0d4427a3240aeda0b57274, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.447350 5661 consensus_queue.cc:260] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.447949 5661 ts_tablet_manager.cc:1428] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 23:03:55.447491 5685 raft_consensus.cc:2229] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:03:55.449057 5685 raft_consensus.cc:2258] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:03:55.454074 5684 ts_tablet_manager.cc:1918] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:03:55.454239 5684 log.cc:1196] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/65d2ca393d3a41a88b0fc268430d2fba
I20260927 23:03:55.454808 5684 ts_tablet_manager.cc:1939] T 65d2ca393d3a41a88b0fc268430d2fba P fc3863d43f0d4427a3240aeda0b57274: Deleting consensus metadata
I20260927 23:03:55.456688 5144 catalog_manager.cc:4841] TS fc3863d43f0d4427a3240aeda0b57274 (127.3.240.194:39957): tablet 65d2ca393d3a41a88b0fc268430d2fba (table test-table [id=3ded869cb0324c1595932bfb6f889f61]) successfully deleted
I20260927 23:03:55.458169 5685 ts_tablet_manager.cc:1905] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:03:55.459695 5662 tablet_bootstrap.cc:492] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: No bootstrap required, opened a new log
I20260927 23:03:55.459844 5662 ts_tablet_manager.cc:1397] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.017s
I20260927 23:03:55.460343 5686 ts_tablet_manager.cc:1905] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:03:55.460666 5662 raft_consensus.cc:348] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.461534 5662 raft_consensus.cc:374] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.461648 5662 raft_consensus.cc:729] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c1d1bbb7f54ba7b1ae64fbe953f9bf, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.461957 5662 consensus_queue.cc:260] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.464496 5662 ts_tablet_manager.cc:1428] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260927 23:03:55.466001 5685 ts_tablet_manager.cc:1918] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:03:55.466176 5685 log.cc:1196] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/65d2ca393d3a41a88b0fc268430d2fba
I20260927 23:03:55.466753 5685 ts_tablet_manager.cc:1939] T 65d2ca393d3a41a88b0fc268430d2fba P a4ed31ff721d4f47827cb8a5647e1def: Deleting consensus metadata
I20260927 23:03:55.467581 5686 ts_tablet_manager.cc:1918] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:03:55.467762 5686 log.cc:1196] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/65d2ca393d3a41a88b0fc268430d2fba
I20260927 23:03:55.468272 5686 ts_tablet_manager.cc:1939] T 65d2ca393d3a41a88b0fc268430d2fba P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Deleting consensus metadata
I20260927 23:03:55.468930 5144 catalog_manager.cc:4841] TS a4ed31ff721d4f47827cb8a5647e1def (127.3.240.195:42147): tablet 65d2ca393d3a41a88b0fc268430d2fba (table test-table [id=3ded869cb0324c1595932bfb6f889f61]) successfully deleted
I20260927 23:03:55.469641 5142 catalog_manager.cc:4841] TS f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525): tablet 65d2ca393d3a41a88b0fc268430d2fba (table test-table [id=3ded869cb0324c1595932bfb6f889f61]) successfully deleted
W20260927 23:03:55.472332 5504 tablet.cc:2369] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:03:55.495721 5365 tablet.cc:2369] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:03:55.560954 5670 raft_consensus.cc:482] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:03:55.561223 5670 raft_consensus.cc:504] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.563663 5670 leader_election.cc:288] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525), a4ed31ff721d4f47827cb8a5647e1def (127.3.240.195:42147)
I20260927 23:03:55.574200 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.564222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:35589 (local address 127.3.240.193:44525)
0927 23:03:55.564410 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.564417 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.567627 (+ 3210us) server_negotiation.cc:407] Connection header received
0927 23:03:55.567921 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.567926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.567998 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.568083 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:55.568930 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.570118 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.572361 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.573204 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.573374 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.573446 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.573772 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.573905 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 23:03:55.574496 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.563930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42147 (local address 127.3.240.194:46221)
0927 23:03:55.567788 (+ 3858us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.567801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.567974 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.568358 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.568364 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.568411 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:55.568841 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.568854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.569914 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.569921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.572055 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.572068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.573758 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.573764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.573909 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.573937 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.574021 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.574131 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3772,"spinlock_wait_cycles":5888,"thread_start_us":3277,"threads_started":1}
I20260927 23:03:55.574622 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.564284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:46221 (local address 127.3.240.195:42147)
0927 23:03:55.564499 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.564508 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.567890 (+ 3382us) server_negotiation.cc:407] Connection header received
0927 23:03:55.568123 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.568129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.568235 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.568335 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:55.568980 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.569772 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.572204 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.573536 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.573700 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.573781 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.574061 (+ 280us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.574201 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 23:03:55.575150 5691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.563887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44525 (local address 127.3.240.194:35589)
0927 23:03:55.567507 (+ 3620us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.567528 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.567751 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.568222 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.568225 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.568242 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:55.568644 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.568658 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.570249 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.570254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.572225 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.572233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.573388 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.573394 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.573576 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.573619 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.574563 (+ 944us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.574704 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3473,"spinlock_wait_cycles":23936,"thread_start_us":3639,"threads_started":1}
I20260927 23:03:55.575481 5592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fc5b475a324795af6d2f291f49547d" candidate_uuid: "fc3863d43f0d4427a3240aeda0b57274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed31ff721d4f47827cb8a5647e1def" is_pre_election: true
I20260927 23:03:55.575858 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fc5b475a324795af6d2f291f49547d" candidate_uuid: "fc3863d43f0d4427a3240aeda0b57274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" is_pre_election: true
I20260927 23:03:55.575996 5592 raft_consensus.cc:2454] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc3863d43f0d4427a3240aeda0b57274 in term 0.
I20260927 23:03:55.576103 5313 raft_consensus.cc:2454] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc3863d43f0d4427a3240aeda0b57274 in term 0.
I20260927 23:03:55.576522 5385 leader_election.cc:302] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6c1d1bbb7f54ba7b1ae64fbe953f9bf, fc3863d43f0d4427a3240aeda0b57274; no voters:
I20260927 23:03:55.576949 5670 raft_consensus.cc:2781] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:03:55.577075 5670 raft_consensus.cc:482] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:03:55.577183 5670 raft_consensus.cc:3037] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.579873 5670 raft_consensus.cc:504] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.580510 5670 leader_election.cc:288] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [CANDIDATE]: Term 1 election: Requested vote from peers f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525), a4ed31ff721d4f47827cb8a5647e1def (127.3.240.195:42147)
I20260927 23:03:55.581168 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fc5b475a324795af6d2f291f49547d" candidate_uuid: "fc3863d43f0d4427a3240aeda0b57274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf"
I20260927 23:03:55.581303 5313 raft_consensus.cc:3037] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.581732 5592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34fc5b475a324795af6d2f291f49547d" candidate_uuid: "fc3863d43f0d4427a3240aeda0b57274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ed31ff721d4f47827cb8a5647e1def"
I20260927 23:03:55.581947 5592 raft_consensus.cc:3037] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.583173 5313 raft_consensus.cc:2454] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc3863d43f0d4427a3240aeda0b57274 in term 1.
I20260927 23:03:55.583478 5385 leader_election.cc:302] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f6c1d1bbb7f54ba7b1ae64fbe953f9bf, fc3863d43f0d4427a3240aeda0b57274; no voters:
I20260927 23:03:55.583719 5670 raft_consensus.cc:2781] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:55.584263 5670 raft_consensus.cc:686] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 1 LEADER]: Becoming Leader. State: Replica: fc3863d43f0d4427a3240aeda0b57274, State: Running, Role: LEADER
I20260927 23:03:55.584647 5670 consensus_queue.cc:237] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.585415 5592 raft_consensus.cc:2454] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc3863d43f0d4427a3240aeda0b57274 in term 1.
I20260927 23:03:55.596411 5157 catalog_manager.cc:5495] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc3863d43f0d4427a3240aeda0b57274 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "fc3863d43f0d4427a3240aeda0b57274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } health_report { overall_health: UNKNOWN } } }
Sep 27 23:03:55 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550233, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:55.666301 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.653362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.3.240.254:34511)
0927 23:03:55.653509 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.653514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.654130 (+ 616us) server_negotiation.cc:407] Connection header received
0927 23:03:55.654306 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.654311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.654391 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.654479 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:55.655625 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.656328 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.657500 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.657932 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.658056 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.661255 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:55.661261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:55.661265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:55.661300 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:55.663825 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.664408 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.664416 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.664419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.664562 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.664882 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.664889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.664892 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.665140 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:55.665274 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.665653 (+ 379us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.666006 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":159,"server-negotiator.queue_time_us":48}
I20260927 23:03:55.666332 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.653490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34511 (local address 127.0.0.1:36070)
0927 23:03:55.654023 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.654037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.654191 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.654506 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.654510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.655218 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:55.655474 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.655483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.656456 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.656463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.657369 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.657378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.658089 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.658094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.658255 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.658855 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:55.658870 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:55.661066 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:55.663935 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:55.663947 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:55.663954 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:55.664306 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:55.664656 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:55.664661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:55.664665 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:55.664790 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:55.665249 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:55.665256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:55.665406 (+ 150us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.665551 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.665885 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":317,"threads_started":1}
I20260927 23:03:55.670948 5157 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36070:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 23:03:55.671949 5157 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:03:55.678779 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 690ed25ce3eb4dc483d504fa94b9c165 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5337ac5585c4568813c89dbbe0ec60f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:03:55.679112 5433 tablet_service.cc:1467] Processing CreateTablet for tablet 690ed25ce3eb4dc483d504fa94b9c165 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5337ac5585c4568813c89dbbe0ec60f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:03:55.679363 5294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690ed25ce3eb4dc483d504fa94b9c165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:55.679680 5433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690ed25ce3eb4dc483d504fa94b9c165. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 23:03:55.681205 5643 tablet.cc:2369] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:03:55.683892 5572 tablet_service.cc:1467] Processing CreateTablet for tablet 690ed25ce3eb4dc483d504fa94b9c165 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b5337ac5585c4568813c89dbbe0ec60f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:03:55.684456 5572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690ed25ce3eb4dc483d504fa94b9c165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:55.685663 5662 tablet_bootstrap.cc:492] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Bootstrap starting.
I20260927 23:03:55.687346 5662 tablet_bootstrap.cc:654] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.687487 5661 tablet_bootstrap.cc:492] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: Bootstrap starting.
I20260927 23:03:55.689602 5661 tablet_bootstrap.cc:654] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.692785 5663 tablet_bootstrap.cc:492] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: Bootstrap starting.
I20260927 23:03:55.695189 5663 tablet_bootstrap.cc:654] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:55.698884 5663 tablet_bootstrap.cc:492] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: No bootstrap required, opened a new log
I20260927 23:03:55.699239 5663 ts_tablet_manager.cc:1397] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260927 23:03:55.700313 5663 raft_consensus.cc:348] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.700595 5663 raft_consensus.cc:374] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.700696 5663 raft_consensus.cc:729] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ed31ff721d4f47827cb8a5647e1def, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.701493 5663 consensus_queue.cc:260] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.701973 5668 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.702231 5661 tablet_bootstrap.cc:492] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: No bootstrap required, opened a new log
I20260927 23:03:55.702258 5668 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:55.702385 5661 ts_tablet_manager.cc:1397] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.012s
I20260927 23:03:55.703181 5661 raft_consensus.cc:348] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.703449 5661 raft_consensus.cc:374] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.703541 5661 raft_consensus.cc:729] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3863d43f0d4427a3240aeda0b57274, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.703855 5661 consensus_queue.cc:260] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.704308 5670 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.704471 5663 ts_tablet_manager.cc:1428] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260927 23:03:55.704483 5661 ts_tablet_manager.cc:1428] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260927 23:03:55.704571 5670 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:55.706096 5662 tablet_bootstrap.cc:492] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: No bootstrap required, opened a new log
I20260927 23:03:55.706278 5662 ts_tablet_manager.cc:1397] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.012s
I20260927 23:03:55.707073 5662 raft_consensus.cc:348] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.707298 5662 raft_consensus.cc:374] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:55.707393 5662 raft_consensus.cc:729] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c1d1bbb7f54ba7b1ae64fbe953f9bf, State: Initialized, Role: FOLLOWER
I20260927 23:03:55.707638 5662 consensus_queue.cc:260] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.708211 5662 ts_tablet_manager.cc:1428] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 23:03:55.708055 5669 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.708441 5669 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:55.776739 5668 raft_consensus.cc:482] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:03:55.777004 5668 raft_consensus.cc:504] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.777691 5668 leader_election.cc:288] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525), fc3863d43f0d4427a3240aeda0b57274 (127.3.240.194:39957)
I20260927 23:03:55.778132 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690ed25ce3eb4dc483d504fa94b9c165" candidate_uuid: "a4ed31ff721d4f47827cb8a5647e1def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" is_pre_election: true
I20260927 23:03:55.778293 5453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690ed25ce3eb4dc483d504fa94b9c165" candidate_uuid: "a4ed31ff721d4f47827cb8a5647e1def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3863d43f0d4427a3240aeda0b57274" is_pre_election: true
I20260927 23:03:55.778411 5313 raft_consensus.cc:2454] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4ed31ff721d4f47827cb8a5647e1def in term 0.
I20260927 23:03:55.778553 5453 raft_consensus.cc:2454] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4ed31ff721d4f47827cb8a5647e1def in term 0.
I20260927 23:03:55.778863 5524 leader_election.cc:302] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ed31ff721d4f47827cb8a5647e1def, f6c1d1bbb7f54ba7b1ae64fbe953f9bf; no voters:
I20260927 23:03:55.779191 5668 raft_consensus.cc:2781] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:03:55.779294 5668 raft_consensus.cc:482] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:03:55.779415 5668 raft_consensus.cc:3037] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.781576 5668 raft_consensus.cc:504] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.782187 5668 leader_election.cc:288] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [CANDIDATE]: Term 1 election: Requested vote from peers f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525), fc3863d43f0d4427a3240aeda0b57274 (127.3.240.194:39957)
I20260927 23:03:55.782512 5313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690ed25ce3eb4dc483d504fa94b9c165" candidate_uuid: "a4ed31ff721d4f47827cb8a5647e1def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf"
I20260927 23:03:55.782749 5313 raft_consensus.cc:3037] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.782696 5453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "690ed25ce3eb4dc483d504fa94b9c165" candidate_uuid: "a4ed31ff721d4f47827cb8a5647e1def" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3863d43f0d4427a3240aeda0b57274"
I20260927 23:03:55.782894 5453 raft_consensus.cc:3037] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:55.785569 5313 raft_consensus.cc:2454] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4ed31ff721d4f47827cb8a5647e1def in term 1.
I20260927 23:03:55.785950 5453 raft_consensus.cc:2454] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4ed31ff721d4f47827cb8a5647e1def in term 1.
I20260927 23:03:55.785961 5524 leader_election.cc:302] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4ed31ff721d4f47827cb8a5647e1def, f6c1d1bbb7f54ba7b1ae64fbe953f9bf; no voters:
I20260927 23:03:55.786216 5668 raft_consensus.cc:2781] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:55.786413 5668 raft_consensus.cc:686] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [term 1 LEADER]: Becoming Leader. State: Replica: a4ed31ff721d4f47827cb8a5647e1def, State: Running, Role: LEADER
I20260927 23:03:55.786688 5668 consensus_queue.cc:237] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } }
I20260927 23:03:55.788455 5667 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: TxnStatusTablet state changed. Reason: New leader a4ed31ff721d4f47827cb8a5647e1def. Latest consensus state: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.788738 5667 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:03:55.789326 5704 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 23:03:55.789568 5704 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 23:03:55.789985 5157 catalog_manager.cc:5495] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def reported cstate change: term changed from 0 to 1, leader changed from <none> to a4ed31ff721d4f47827cb8a5647e1def (127.3.240.195). New cstate: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } health_report { overall_health: HEALTHY } } }
I20260927 23:03:55.812011 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.807240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.3.240.195:42147)
0927 23:03:55.807435 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.807441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.807543 (+ 102us) server_negotiation.cc:407] Connection header received
0927 23:03:55.807780 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.807785 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.807853 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.807936 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:03:55.808298 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.808997 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.810205 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.810670 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.810839 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.811328 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:03:55.811419 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:03:55.811526 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.811637 (+ 111us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.811709 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 23:03:55.812029 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.807233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42147 (local address 127.0.0.1:33744)
0927 23:03:55.807452 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.807472 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.807627 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.807981 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.807996 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.808008 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:03:55.808178 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.808186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.809137 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.809143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.810078 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.810088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.810848 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.810853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.811106 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.811162 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:03:55.811526 (+ 364us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:03:55.811546 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.811620 (+ 74us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.811753 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 23:03:55.815567 5453 raft_consensus.cc:1261] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Refusing update from remote peer a4ed31ff721d4f47827cb8a5647e1def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:03:55.815632 5313 raft_consensus.cc:1261] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Refusing update from remote peer a4ed31ff721d4f47827cb8a5647e1def: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:03:55.816176 5667 consensus_queue.cc:1035] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:03:55.816430 5668 consensus_queue.cc:1035] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:03:55.820545 5670 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: TxnStatusTablet state changed. Reason: New leader a4ed31ff721d4f47827cb8a5647e1def. Latest consensus state: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.820776 5670 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:55.821842 5667 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.822094 5667 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:03:55.823812 5676 mvcc.cc:205] Tried to move back new op lower bound from 7334093765895086080 to 7334093765784580096. Current Snapshot: MvccSnapshot[applied={T|T < 7334093765895086080}]
I20260927 23:03:55.824153 5670 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.824414 5670 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P fc3863d43f0d4427a3240aeda0b57274: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:55.824587 5669 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: TxnStatusTablet state changed. Reason: New leader a4ed31ff721d4f47827cb8a5647e1def. Latest consensus state: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.824822 5669 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:55.827260 5667 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.827515 5667 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P a4ed31ff721d4f47827cb8a5647e1def: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:03:55.831532 5669 tablet_replica.cc:439] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4ed31ff721d4f47827cb8a5647e1def" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 } } peers { permanent_uuid: "fc3863d43f0d4427a3240aeda0b57274" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39957 } } peers { permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 } } }
I20260927 23:03:55.831755 5669 tablet_replica.cc:442] T 690ed25ce3eb4dc483d504fa94b9c165 P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:55.837623 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.829828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.3.240.194:39957)
0927 23:03:55.832456 (+ 2628us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.832461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.832478 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:03:55.832563 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.832570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.832659 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.833026 (+ 367us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:03:55.833368 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.834168 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.835538 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.835962 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.836108 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.836555 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:03:55.837063 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:03:55.837207 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.837263 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.837336 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2528,"thread_start_us":3587,"threads_started":1}
I20260927 23:03:55.853363 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.847420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:46335 (local address 127.3.240.195:42147)
0927 23:03:55.847628 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.847636 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.848040 (+ 404us) server_negotiation.cc:407] Connection header received
0927 23:03:55.848178 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.848181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.848234 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.848299 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:03:55.848781 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.849549 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.851819 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.852246 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.852347 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.852791 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:03:55.852903 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:03:55.852963 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.853104 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.853165 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:03:55.853572 5711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.847390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42147 (local address 127.3.240.194:46335)
0927 23:03:55.847950 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.847967 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.848083 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.848333 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.848338 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.848349 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:03:55.848661 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.848671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.849669 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.849676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.851699 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.851712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.852422 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.852428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.852587 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.852648 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:03:55.852994 (+ 346us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:03:55.853005 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.853081 (+ 76us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.853219 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":296,"threads_started":1}
I20260927 23:03:55.862191 5313 raft_consensus.cc:1261] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Refusing update from remote peer fc3863d43f0d4427a3240aeda0b57274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:03:55.862484 5592 raft_consensus.cc:1261] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Refusing update from remote peer fc3863d43f0d4427a3240aeda0b57274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:03:55.863168 5692 consensus_queue.cc:1035] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ed31ff721d4f47827cb8a5647e1def" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:03:55.863492 5670 consensus_queue.cc:1035] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c1d1bbb7f54ba7b1ae64fbe953f9bf" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:03:55.900223 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.894033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:51605 (local address 127.3.240.194:39957)
0927 23:03:55.894434 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.894440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.894876 (+ 436us) server_negotiation.cc:407] Connection header received
0927 23:03:55.895140 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.895147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.895228 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.895320 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:03:55.895909 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.896661 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.898406 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.899034 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.899154 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.899698 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:03:55.899789 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:03:55.899869 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.899991 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.900044 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260927 23:03:55.900506 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.894152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:39957 (local address 127.3.240.195:51605)
0927 23:03:55.894774 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:55.894795 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:55.894962 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:55.895350 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:55.895355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:55.895368 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:03:55.895740 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.895750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.896785 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.896790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.898271 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:55.898284 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.899172 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.899296 (+ 124us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:55.899476 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.899559 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:03:55.899888 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:03:55.899904 (+ 16us) client_negotiation.cc:769] Sending connection context
0927 23:03:55.899981 (+ 77us) client_negotiation.cc:240] Negotiation successful
0927 23:03:55.900114 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":293,"threads_started":1}
I20260927 23:03:55.935178 5156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:03:55.935425 5156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:03:55.938571 5156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a64c0119b1624364b6337f74500a6e7c: Sending DeleteTablet for 3 replicas of tablet 34fc5b475a324795af6d2f291f49547d
I20260927 23:03:55.939284 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fc5b475a324795af6d2f291f49547d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:03:55 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:35228
I20260927 23:03:55.939563 5686 tablet_replica.cc:330] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: stopping tablet replica
I20260927 23:03:55.939620 5433 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fc5b475a324795af6d2f291f49547d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:03:55 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57884
I20260927 23:03:55.939749 5686 raft_consensus.cc:2229] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:03:55.939864 5684 tablet_replica.cc:330] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: stopping tablet replica
I20260927 23:03:55.939922 5686 raft_consensus.cc:2258] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:03:55.940065 5684 raft_consensus.cc:2229] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:03:55.940296 5572 tablet_service.cc:1514] Processing DeleteTablet for tablet 34fc5b475a324795af6d2f291f49547d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:03:55 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:33728
I20260927 23:03:55.940587 5684 raft_consensus.cc:2258] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:03:55.940869 5685 tablet_replica.cc:330] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: stopping tablet replica
I20260927 23:03:55.941056 5685 raft_consensus.cc:2229] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:03:55.941250 5685 raft_consensus.cc:2258] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:03:55.941898 5684 ts_tablet_manager.cc:1905] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:03:55.942386 5685 ts_tablet_manager.cc:1905] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:03:55.949048 5684 ts_tablet_manager.cc:1918] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:03:55.949186 5684 log.cc:1196] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/34fc5b475a324795af6d2f291f49547d
I20260927 23:03:55.949625 5684 ts_tablet_manager.cc:1939] T 34fc5b475a324795af6d2f291f49547d P fc3863d43f0d4427a3240aeda0b57274: Deleting consensus metadata
I20260927 23:03:55.949908 5685 ts_tablet_manager.cc:1918] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:03:55.950078 5685 log.cc:1196] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/34fc5b475a324795af6d2f291f49547d
I20260927 23:03:55.950441 5685 ts_tablet_manager.cc:1939] T 34fc5b475a324795af6d2f291f49547d P a4ed31ff721d4f47827cb8a5647e1def: Deleting consensus metadata
I20260927 23:03:55.950651 5144 catalog_manager.cc:4841] TS fc3863d43f0d4427a3240aeda0b57274 (127.3.240.194:39957): tablet 34fc5b475a324795af6d2f291f49547d (table test-table [id=c5320657d28e4f64a734291365b9d7d8]) successfully deleted
I20260927 23:03:55.952797 5686 ts_tablet_manager.cc:1905] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:03:55.959924 5686 ts_tablet_manager.cc:1918] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:03:55.960088 5686 log.cc:1196] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/34fc5b475a324795af6d2f291f49547d
I20260927 23:03:55.960556 5686 ts_tablet_manager.cc:1939] T 34fc5b475a324795af6d2f291f49547d P f6c1d1bbb7f54ba7b1ae64fbe953f9bf: Deleting consensus metadata
I20260927 23:03:55.961928 5142 catalog_manager.cc:4841] TS f6c1d1bbb7f54ba7b1ae64fbe953f9bf (127.3.240.193:44525): tablet 34fc5b475a324795af6d2f291f49547d (table test-table [id=c5320657d28e4f64a734291365b9d7d8]) successfully deleted
I20260927 23:03:55.962761 5144 catalog_manager.cc:4841] TS a4ed31ff721d4f47827cb8a5647e1def (127.3.240.195:42147): tablet 34fc5b475a324795af6d2f291f49547d (table test-table [id=c5320657d28e4f64a734291365b9d7d8]) successfully deleted
Sep 27 23:03:55 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:03:55.982134 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.959179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.3.240.193:44525)
0927 23:03:55.959374 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.959382 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.963680 (+ 4298us) server_negotiation.cc:407] Connection header received
0927 23:03:55.963873 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.963880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.963970 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.964074 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:55.965944 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.966972 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.968934 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.969572 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.969694 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:55.975118 (+ 5424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:55.975134 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:55.975182 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:55.975223 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:55.978816 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.979584 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.979594 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.979598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.979744 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:55.980131 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:55.980136 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:55.980140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:55.980548 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:55.980829 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:55.981527 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 23:03:55.981733 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 23:03:55.982888 5359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35244
I20260927 23:03:56.006906 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:55.988089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.3.240.254:34511)
0927 23:03:55.988268 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:55.988275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:55.991479 (+ 3204us) server_negotiation.cc:407] Connection header received
0927 23:03:55.991636 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:55.991641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:55.991722 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:55.991831 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:55.993825 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.994953 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.996767 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:55.997506 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:55.997602 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:56.000765 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:56.000771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:56.000776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:56.000815 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:56.004211 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:56.004909 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:56.004915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:56.004918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:56.005052 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:56.005428 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:56.005433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:56.005437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:56.005762 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:56.006025 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:56.006298 (+ 273us) server_negotiation.cc:299] Negotiation successful
0927 23:03:56.006512 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 23:03:56.007622 5156 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36076
I20260927 23:03:56.009824 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5228
I20260927 23:03:56.034934 5658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:56.011449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.3.240.193:44525)
0927 23:03:56.011610 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:56.011616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:56.019792 (+ 8176us) server_negotiation.cc:407] Connection header received
0927 23:03:56.019967 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:56.019974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:56.020064 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:56.020186 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:56.021637 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:56.022768 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:56.024579 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:56.025240 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:56.025390 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:56.028212 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:56.028220 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:56.028225 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:56.028266 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:56.031957 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:56.032758 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:56.032766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:56.032772 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:56.032929 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:56.033338 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:56.033344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:56.033348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:56.033691 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:56.033965 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:56.034274 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 23:03:56.034505 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260927 23:03:56.035804 5359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35246
W20260927 23:03:56.036365 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5228: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:03:56.036576 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 5228
I20260927 23:03:56.061633 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5368
Sep 27 23:03:56 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:03:56.081552 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:56.062451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57898 (local address 127.3.240.194:39957)
0927 23:03:56.062686 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:56.062692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:56.062986 (+ 294us) server_negotiation.cc:407] Connection header received
0927 23:03:56.063222 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:56.063229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:56.063375 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:56.063511 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:56.065097 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:56.066288 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:56.068095 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:56.068805 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:56.068968 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:56.073974 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:56.073990 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:56.074010 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:56.074058 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:56.078510 (+ 4452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:56.079306 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:56.079316 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:56.079320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:56.079519 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:56.079940 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:56.079945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:56.079949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:56.080388 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:56.080704 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:56.080941 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 23:03:56.081136 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 23:03:56.082209 5498 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57898
W20260927 23:03:56.082916 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5368: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:03:56.083130 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 5368
I20260927 23:03:56.101621 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5507
Sep 27 23:03:56 dist-test-slave-lmfp krb5kdc[5112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550235, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:03:56.122238 5659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:56.102293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.3.240.195:42147)
0927 23:03:56.102456 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:56.102460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:56.102509 (+ 49us) server_negotiation.cc:407] Connection header received
0927 23:03:56.102731 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:56.102735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:56.102798 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:56.102871 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:56.104343 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:56.105066 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:56.106801 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:56.107279 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:56.107378 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:56.111838 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:56.111861 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:56.111886 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:56.111921 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:56.119695 (+ 7774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:56.120257 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:56.120265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:56.120269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:56.120393 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:56.120751 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:56.120756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:56.120759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:56.121161 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:56.121453 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:56.121596 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 23:03:56.121854 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260927 23:03:56.122880 5637 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33756
W20260927 23:03:56.123520 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5507: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:03:56.123670 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 5507
I20260927 23:03:56.144840 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5125
I20260927 23:03:56.145876 5197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:03:56.256196 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 5125
2026-09-27T23:03:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4229 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:03:56 dist-test-slave-lmfp krb5kdc[5727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:03:56 dist-test-slave-lmfp krb5kdc[5727](info): set up 2 sockets
Sep 27 23:03:56 dist-test-slave-lmfp krb5kdc[5727](info): commencing operation
krb5kdc: starting...
W20260927 23:03:57.327391 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:03:57 dist-test-slave-lmfp krb5kdc[5727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:03:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:03:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:03:57.506033 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45487
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:57.820331 5740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:57.820644 5740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:57.820782 5740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:57.827765 5740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:03:57.827890 5740 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 23:03:57.827986 5740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:57.828094 5740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:57.828183 5740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:03:57.828274 5740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:03:57.837792 5740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45487
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:57.839056 5740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:57.840395 5740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:57.852028 5745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:57.851933 5748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:57.851995 5746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:57.852764 5740 server_base.cc:1034] running on GCE node
I20260927 23:03:57.853416 5740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:57.854631 5740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:57.855839 5740 hybrid_clock.cc:648] HybridClock initialized: now 1790550237855748 us; error 50 us; skew 500 ppm
I20260927 23:03:57.856112 5740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:57 dist-test-slave-lmfp krb5kdc[5727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550237, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:57.859156 5740 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:03:57.860534 5740 webserver.cc:460] Webserver started at http://127.3.240.254:34021/ using document root <none> and password file <none>
I20260927 23:03:57.861177 5740 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:57.861258 5740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:57.861495 5740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:57.864285 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "a05984665a814393b262309e52a01252"
format_stamp: "Formatted at 2026-09-27 23:03:57 on dist-test-slave-lmfp"
server_key: "cc11ddbd721a81aa920811bc281d3401"
server_key_iv: "e6607465e6b76d7a49a07cd99b4e7893"
server_key_version: "encryptionkey@0"
I20260927 23:03:57.864946 5740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "a05984665a814393b262309e52a01252"
format_stamp: "Formatted at 2026-09-27 23:03:57 on dist-test-slave-lmfp"
server_key: "cc11ddbd721a81aa920811bc281d3401"
server_key_iv: "e6607465e6b76d7a49a07cd99b4e7893"
server_key_version: "encryptionkey@0"
I20260927 23:03:57.869751 5740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 23:03:57.872936 5755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:57.874230 5740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:03:57.874394 5740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "a05984665a814393b262309e52a01252"
format_stamp: "Formatted at 2026-09-27 23:03:57 on dist-test-slave-lmfp"
server_key: "cc11ddbd721a81aa920811bc281d3401"
server_key_iv: "e6607465e6b76d7a49a07cd99b4e7893"
server_key_version: "encryptionkey@0"
I20260927 23:03:57.874590 5740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:57.892572 5740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:57.896728 5740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:57.896993 5740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:57.918408 5740 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:45487
I20260927 23:03:57.918416 5816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:45487 every 8 connection(s)
I20260927 23:03:57.919809 5740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:03:57.921832 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 5740
I20260927 23:03:57.922052 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:03:57.922850 4035 external_mini_cluster.cc:1428] Setting key e63bf7975830ab80b8223b9602371e2b
I20260927 23:03:57.924437 5817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 23:03:57 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:57.938360 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: Bootstrap starting.
I20260927 23:03:57.941509 5817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:57.942476 5817 log.cc:824] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:57.945143 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:57.924902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.3.240.254:45487)
0927 23:03:57.925674 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:57.925686 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:57.925727 (+ 41us) server_negotiation.cc:407] Connection header received
0927 23:03:57.927352 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:57.927373 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:57.927778 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:57.928115 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:57.929887 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:57.931314 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:57.933046 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:57.933696 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:57.933819 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:57.938391 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:57.938399 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:57.938412 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:57.938473 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:57.941410 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:57.942086 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:57.942093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:57.942099 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:57.942247 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:57.942606 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:57.942614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:57.942619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:57.943189 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:57.943526 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:57.943861 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 23:03:57.944246 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":366,"threads_started":1}
I20260927 23:03:57.945351 5817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: No bootstrap required, opened a new log
I20260927 23:03:57.953027 5817 raft_consensus.cc:348] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05984665a814393b262309e52a01252" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45487 } }
I20260927 23:03:57.953401 5817 raft_consensus.cc:374] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:57.953507 5817 raft_consensus.cc:729] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a05984665a814393b262309e52a01252, State: Initialized, Role: FOLLOWER
I20260927 23:03:57.954032 5817 consensus_queue.cc:260] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05984665a814393b262309e52a01252" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45487 } }
I20260927 23:03:57.954252 5817 raft_consensus.cc:388] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:03:57.954384 5817 raft_consensus.cc:482] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:03:57.954555 5817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:57.956776 5817 raft_consensus.cc:504] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05984665a814393b262309e52a01252" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45487 } }
I20260927 23:03:57.957255 5817 leader_election.cc:302] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a05984665a814393b262309e52a01252; no voters:
I20260927 23:03:57.958216 5817 leader_election.cc:288] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:03:57.958426 5822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:57.959223 5822 raft_consensus.cc:686] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [term 1 LEADER]: Becoming Leader. State: Replica: a05984665a814393b262309e52a01252, State: Running, Role: LEADER
I20260927 23:03:57.959745 5822 consensus_queue.cc:237] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05984665a814393b262309e52a01252" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45487 } }
I20260927 23:03:57.960315 5817 sys_catalog.cc:564] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:03:57.962862 5824 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a05984665a814393b262309e52a01252. Latest consensus state: current_term: 1 leader_uuid: "a05984665a814393b262309e52a01252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05984665a814393b262309e52a01252" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45487 } } }
I20260927 23:03:57.963302 5824 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [sys.catalog]: This master's current role is: LEADER
I20260927 23:03:57.962849 5823 sys_catalog.cc:455] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a05984665a814393b262309e52a01252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a05984665a814393b262309e52a01252" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45487 } } }
I20260927 23:03:57.965240 5823 sys_catalog.cc:458] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252 [sys.catalog]: This master's current role is: LEADER
I20260927 23:03:57.966786 5829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:03:57.972033 5829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:03:57.980713 5829 catalog_manager.cc:1347] Generated new cluster ID: 04470aab1dd544ab96946ef291582d31
I20260927 23:03:57.980844 5829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:03:57.998222 5829 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:03:57.999585 5829 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:03:58.006419 5829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: Generated new TSK 0
I20260927 23:03:58.007232 5829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:03:58.052469 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45487
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:58.371145 5843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:58.371385 5843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:58.371547 5843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:58.378305 5843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:58.378400 5843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:58.378569 5843 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 23:03:58.378659 5843 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:03:58.378762 5843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:03:58.387847 5843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:45487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:58.389031 5843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:58.390370 5843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:58.402441 5849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:58.402382 5851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:58.403107 5848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:58.403092 5843 server_base.cc:1034] running on GCE node
I20260927 23:03:58.403775 5843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:58.404618 5843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:58.405797 5843 hybrid_clock.cc:648] HybridClock initialized: now 1790550238405748 us; error 48 us; skew 500 ppm
I20260927 23:03:58.406062 5843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:58 dist-test-slave-lmfp krb5kdc[5727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550238, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:58.408929 5843 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:03:58.410147 5843 webserver.cc:460] Webserver started at http://127.3.240.193:39517/ using document root <none> and password file <none>
I20260927 23:03:58.410748 5843 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:58.410867 5843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:58.411183 5843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:58.413654 5843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "0c725c7ecf4944cab607f1eb06de1772"
format_stamp: "Formatted at 2026-09-27 23:03:58 on dist-test-slave-lmfp"
server_key: "ab37def8b050ba4dcb9778f8a0b881c7"
server_key_iv: "eebe1f191ab5c76a5bbefaaf64fb50fe"
server_key_version: "encryptionkey@0"
I20260927 23:03:58.414321 5843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "0c725c7ecf4944cab607f1eb06de1772"
format_stamp: "Formatted at 2026-09-27 23:03:58 on dist-test-slave-lmfp"
server_key: "ab37def8b050ba4dcb9778f8a0b881c7"
server_key_iv: "eebe1f191ab5c76a5bbefaaf64fb50fe"
server_key_version: "encryptionkey@0"
I20260927 23:03:58.418929 5843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 23:03:58.422039 5858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:58.423326 5843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260927 23:03:58.423480 5843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "0c725c7ecf4944cab607f1eb06de1772"
format_stamp: "Formatted at 2026-09-27 23:03:58 on dist-test-slave-lmfp"
server_key: "ab37def8b050ba4dcb9778f8a0b881c7"
server_key_iv: "eebe1f191ab5c76a5bbefaaf64fb50fe"
server_key_version: "encryptionkey@0"
I20260927 23:03:58.423682 5843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:58.436092 5843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:58.439585 5843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:58.439839 5843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:58.442227 5843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:58.442340 5843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:58.442444 5843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:58.442529 5843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 23:03:58 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550238, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:58.466229 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:58.443020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:53751 (local address 127.3.240.254:45487)
0927 23:03:58.443179 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:58.443184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:58.444729 (+ 1545us) server_negotiation.cc:407] Connection header received
0927 23:03:58.446666 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:58.446672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:58.446754 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:58.446870 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:58.448897 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.449909 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.452699 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.453147 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.453288 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:58.459176 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:58.459184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:58.459191 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:58.459238 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:58.462439 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:58.463610 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:58.463616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:58.463620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:58.463791 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:58.464358 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:58.464364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:58.464367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:58.464810 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:58.464959 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:58.465660 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 23:03:58.465882 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260927 23:03:58.467715 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:58.443646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45487 (local address 127.3.240.193:53751)
0927 23:03:58.444545 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:58.444608 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:58.446405 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:58.447142 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:58.447156 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:58.447780 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:58.448716 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:58.448740 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.450141 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.450148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:58.452473 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:58.452503 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.453449 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.453460 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:58.453815 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:58.454726 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:58.454773 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:58.458762 (+ 3989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:58.462584 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:58.462598 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:58.462626 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:58.463404 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:58.463961 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:58.463971 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:58.463975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:58.464212 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:58.464957 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:58.464965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:58.465309 (+ 344us) client_negotiation.cc:769] Sending connection context
0927 23:03:58.465560 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 23:03:58.466099 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":332,"threads_started":1}
I20260927 23:03:58.487047 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:58.473470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:42471 (local address 127.3.240.254:45487)
0927 23:03:58.473611 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:58.473615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:58.474247 (+ 632us) server_negotiation.cc:407] Connection header received
0927 23:03:58.474461 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:58.474464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:58.474517 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:58.474607 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:58.475873 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.476734 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.478151 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.478662 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.478783 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:58.481060 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:58.481066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:58.481070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:58.481106 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:58.483696 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:58.484668 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:58.484674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:58.484678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:58.484836 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:58.485275 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:58.485282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:58.485286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:58.485676 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:58.485825 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:58.486463 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 23:03:58.486671 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:03:58.487314 5929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:58.473560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45487 (local address 127.3.240.193:42471)
0927 23:03:58.474147 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:58.474165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:58.474325 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:58.474638 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:58.474645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:58.475266 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:58.475735 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:58.475746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.476861 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.476869 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:58.478018 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:58.478030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.478845 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.478854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:58.479089 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:58.479768 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:58.479797 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:58.480876 (+ 1079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:58.483829 (+ 2953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:58.483834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:58.483841 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:58.484471 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:58.484973 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:58.484979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:58.484989 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:58.485163 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:58.485880 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:58.485884 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:58.486177 (+ 293us) client_negotiation.cc:769] Sending connection context
0927 23:03:58.486373 (+ 196us) client_negotiation.cc:240] Negotiation successful
0927 23:03:58.486688 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":340,"threads_started":1}
I20260927 23:03:58.495369 5843 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:37123
I20260927 23:03:58.495404 5977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:37123 every 8 connection(s)
I20260927 23:03:58.496495 5843 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:03:58.498315 5978 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45487
I20260927 23:03:58.498529 5978 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:58.499208 5978 heartbeater.cc:511] Master 127.3.240.254:45487 requested a full tablet report, sending...
I20260927 23:03:58.500931 5772 ts_manager.cc:194] Registered new tserver with Master: 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123)
I20260927 23:03:58.501245 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 5843
I20260927 23:03:58.501420 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:03:58.502028 4035 external_mini_cluster.cc:1428] Setting key 811df4d29a7a9067e1bd52d28a92abed
I20260927 23:03:58.502421 5772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:53751
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:03:58.537817 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45487
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:58.845449 5982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:58.845712 5982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:58.845804 5982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:58.852684 5982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:58.852782 5982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:58.852877 5982 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 23:03:58.852942 5982 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:03:58.853016 5982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:03:58.861968 5982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:45487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:58.863161 5982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:58.864452 5982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:58.876924 5987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:58.876981 5988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:58.876988 5990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:58.877753 5982 server_base.cc:1034] running on GCE node
I20260927 23:03:58.878407 5982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:58.879352 5982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:58.880563 5982 hybrid_clock.cc:648] HybridClock initialized: now 1790550238880523 us; error 46 us; skew 500 ppm
I20260927 23:03:58.880851 5982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:58 dist-test-slave-lmfp krb5kdc[5727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550238, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:58.883733 5982 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:03:58.884935 5982 webserver.cc:460] Webserver started at http://127.3.240.194:42285/ using document root <none> and password file <none>
I20260927 23:03:58.885479 5982 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:58.885565 5982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:58.885788 5982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:58.888401 5982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "fd5ac701368640d7a33a2a9c550872f8"
format_stamp: "Formatted at 2026-09-27 23:03:58 on dist-test-slave-lmfp"
server_key: "8db148e3011ca36ad5340cf441fddd28"
server_key_iv: "5db8dbe64af1babb1fb3dcc869d512b6"
server_key_version: "encryptionkey@0"
I20260927 23:03:58.889045 5982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "fd5ac701368640d7a33a2a9c550872f8"
format_stamp: "Formatted at 2026-09-27 23:03:58 on dist-test-slave-lmfp"
server_key: "8db148e3011ca36ad5340cf441fddd28"
server_key_iv: "5db8dbe64af1babb1fb3dcc869d512b6"
server_key_version: "encryptionkey@0"
I20260927 23:03:58.893821 5982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260927 23:03:58.896705 5997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:58.897835 5982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:03:58.897943 5982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "fd5ac701368640d7a33a2a9c550872f8"
format_stamp: "Formatted at 2026-09-27 23:03:58 on dist-test-slave-lmfp"
server_key: "8db148e3011ca36ad5340cf441fddd28"
server_key_iv: "5db8dbe64af1babb1fb3dcc869d512b6"
server_key_version: "encryptionkey@0"
I20260927 23:03:58.898087 5982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:58.911772 5982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:58.915098 5982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:58.915314 5982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:58.917711 5982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:58.917826 5982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:58.918004 5982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:58.918138 5982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 27 23:03:58 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550238, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:58.940742 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:58.918287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:50081 (local address 127.3.240.254:45487)
0927 23:03:58.918527 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:58.918536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:58.919882 (+ 1346us) server_negotiation.cc:407] Connection header received
0927 23:03:58.922735 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:58.922741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:58.922814 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:58.922903 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:58.924911 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.926005 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.928498 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.929081 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.929162 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:58.933788 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:58.933795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:58.933799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:58.933836 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:58.937188 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:58.938105 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:58.938110 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:58.938114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:58.938233 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:58.938724 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:58.938729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:58.938732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:58.939132 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:58.939299 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:58.940171 (+ 872us) server_negotiation.cc:299] Negotiation successful
0927 23:03:58.940398 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:03:58.942217 6006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:58.918715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45487 (local address 127.3.240.194:50081)
0927 23:03:58.919688 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:58.919748 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:58.922400 (+ 2652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:58.923280 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:58.923290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:58.923982 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:58.924729 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:58.924752 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.926222 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.926226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:58.928354 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:58.928369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.929478 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.929484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:58.929736 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:58.930425 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:58.930449 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:58.933528 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:58.937346 (+ 3818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:58.937355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:58.937369 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:58.937977 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:58.938344 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:58.938348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:58.938351 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:58.938618 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:58.939295 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:58.939302 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:58.939776 (+ 474us) client_negotiation.cc:769] Sending connection context
0927 23:03:58.940081 (+ 305us) client_negotiation.cc:240] Negotiation successful
0927 23:03:58.940589 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":414,"threads_started":1}
I20260927 23:03:58.963064 5982 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:46785
I20260927 23:03:58.963122 6116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:46785 every 8 connection(s)
I20260927 23:03:58.964005 6073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:58.948061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45487 (local address 127.3.240.194:51113)
0927 23:03:58.949312 (+ 1251us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:58.949336 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:58.949488 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:58.949823 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:58.949827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:58.950234 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:58.950495 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:58.950503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.951852 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.951860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:58.954035 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:58.954053 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.955013 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.955024 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:58.955226 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:58.955785 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:58.955811 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:58.956999 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:58.961239 (+ 4240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:58.961243 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:58.961250 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:58.961736 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:58.962121 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:58.962129 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:58.962132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:58.962307 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:58.963104 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:58.963114 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:58.963374 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 23:03:58.963492 (+ 118us) client_negotiation.cc:240] Negotiation successful
0927 23:03:58.963714 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":680,"threads_started":1}
I20260927 23:03:58.964128 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:58.948447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:51113 (local address 127.3.240.254:45487)
0927 23:03:58.948668 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:58.948676 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:58.949425 (+ 749us) server_negotiation.cc:407] Connection header received
0927 23:03:58.949620 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:58.949625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:58.949693 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:58.949797 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:58.950619 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.951682 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.954185 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:58.954784 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:58.954923 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:58.957217 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:58.957224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:58.957228 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:58.957265 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:58.961047 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:58.961839 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:58.961842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:58.961844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:58.961999 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:58.962464 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:58.962472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:58.962477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:58.962905 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:58.963068 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:58.963602 (+ 534us) server_negotiation.cc:299] Negotiation successful
0927 23:03:58.963812 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 23:03:58.964560 5982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:03:58.966487 6117 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45487
I20260927 23:03:58.966691 6117 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:58.967489 6117 heartbeater.cc:511] Master 127.3.240.254:45487 requested a full tablet report, sending...
I20260927 23:03:58.968677 5771 ts_manager.cc:194] Registered new tserver with Master: fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785)
I20260927 23:03:58.969475 5771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:50081
I20260927 23:03:58.973585 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 5982
I20260927 23:03:58.973749 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:03:58.974181 4035 external_mini_cluster.cc:1428] Setting key a79b62c92b368940ff1e26de6bd7f702
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:03:59.009687 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45487
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:03:59.314219 6121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:03:59.314483 6121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:03:59.314637 6121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:03:59.321135 6121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:03:59.321228 6121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:03:59.321378 6121 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260927 23:03:59.321455 6121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:03:59.321588 6121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:03:59.330879 6121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:45487
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:03:59.332067 6121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:03:59.333324 6121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:03:59.345742 6126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:59.345665 6127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:03:59.345810 6129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:03:59.346680 6121 server_base.cc:1034] running on GCE node
I20260927 23:03:59.347283 6121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:03:59.348136 6121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:03:59.349354 6121 hybrid_clock.cc:648] HybridClock initialized: now 1790550239349298 us; error 53 us; skew 500 ppm
I20260927 23:03:59.349639 6121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:03:59 dist-test-slave-lmfp krb5kdc[5727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550239, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:03:59.352427 6121 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:03:59.353735 6121 webserver.cc:460] Webserver started at http://127.3.240.195:37793/ using document root <none> and password file <none>
I20260927 23:03:59.354348 6121 fs_manager.cc:359] Metadata directory not provided
I20260927 23:03:59.354493 6121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:03:59.354774 6121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:03:59.357255 6121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf"
format_stamp: "Formatted at 2026-09-27 23:03:59 on dist-test-slave-lmfp"
server_key: "0a1b070109914968b44b6152e07b7e0d"
server_key_iv: "f353623fe6ac8ef118fdc6f30a87bf4e"
server_key_version: "encryptionkey@0"
I20260927 23:03:59.357944 6121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf"
format_stamp: "Formatted at 2026-09-27 23:03:59 on dist-test-slave-lmfp"
server_key: "0a1b070109914968b44b6152e07b7e0d"
server_key_iv: "f353623fe6ac8ef118fdc6f30a87bf4e"
server_key_version: "encryptionkey@0"
I20260927 23:03:59.362511 6121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 23:03:59.365676 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:59.366855 6121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:03:59.367067 6121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf"
format_stamp: "Formatted at 2026-09-27 23:03:59 on dist-test-slave-lmfp"
server_key: "0a1b070109914968b44b6152e07b7e0d"
server_key_iv: "f353623fe6ac8ef118fdc6f30a87bf4e"
server_key_version: "encryptionkey@0"
I20260927 23:03:59.367252 6121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:03:59.404007 6121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:03:59.407683 6121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:03:59.407955 6121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:03:59.410375 6121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:03:59.410492 6121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:03:59.410619 6121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:03:59.410751 6121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 23:03:59 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550239, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:59.436125 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.410766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:46611 (local address 127.3.240.254:45487)
0927 23:03:59.410980 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.411002 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.413130 (+ 2128us) server_negotiation.cc:407] Connection header received
0927 23:03:59.416280 (+ 3150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.416287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.416371 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.416496 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:59.418697 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.420020 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.422294 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.422846 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.422972 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.428411 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:59.428420 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:59.428426 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:59.428470 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:59.432048 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:59.433029 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:59.433038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:59.433044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:59.433180 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:59.433697 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:59.433704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:59.433709 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:59.434062 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:59.434221 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.435267 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.435609 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 23:03:59.436764 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.412157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45487 (local address 127.3.240.195:46611)
0927 23:03:59.412962 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.413016 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.415893 (+ 2877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.416793 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.416807 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.417660 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:59.418463 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.418503 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.420197 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.420206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.422086 (+ 1880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.422103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.423124 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.423135 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.423467 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.424237 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:59.424275 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:59.428142 (+ 3867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:59.432223 (+ 4081us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:59.432235 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:59.432277 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:59.432903 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:59.433300 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:59.433305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:59.433309 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:59.433576 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:59.434214 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:59.434229 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:59.434760 (+ 531us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.435145 (+ 385us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.435569 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":332,"threads_started":1}
I20260927 23:03:59.456930 6121 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:45645
I20260927 23:03:59.457002 6255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:45645 every 8 connection(s)
I20260927 23:03:59.458248 6121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:03:59.460561 6256 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45487
I20260927 23:03:59.460794 6256 heartbeater.cc:464] Registering TS with master...
I20260927 23:03:59.461297 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.447210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:43165 (local address 127.3.240.254:45487)
0927 23:03:59.447365 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.447369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.448055 (+ 686us) server_negotiation.cc:407] Connection header received
0927 23:03:59.448348 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.448356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.448443 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.448558 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:59.450176 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.451231 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.452519 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.453228 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.453360 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.455558 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:59.455563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:59.455567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:59.455597 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:59.458547 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:59.459202 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:59.459207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:59.459210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:59.459405 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:59.459828 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:59.459835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:59.459841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:59.460182 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:59.460327 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.460765 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.460972 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:03:59.461315 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.447283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45487 (local address 127.3.240.195:43165)
0927 23:03:59.447908 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.447949 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.448181 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.448627 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.448635 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.449383 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:59.450001 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.450017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.451358 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.451364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.452389 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.452399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.453458 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.453468 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.453709 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.454470 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:59.454497 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:59.455405 (+ 908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:59.458658 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:59.458665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:59.458669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:59.459078 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:59.459560 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:59.459565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:59.459567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:59.459720 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:59.460298 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:59.460303 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:59.460479 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.460620 (+ 141us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.460929 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":333,"threads_started":1}
I20260927 23:03:59.461627 6256 heartbeater.cc:511] Master 127.3.240.254:45487 requested a full tablet report, sending...
I20260927 23:03:59.462954 5771 ts_manager.cc:194] Registered new tserver with Master: fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:03:59.463912 5771 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:46611
I20260927 23:03:59.464335 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 6121
I20260927 23:03:59.464510 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:03:59.464983 4035 external_mini_cluster.cc:1428] Setting key 20312d2b23bb63429e614b78ca515427
I20260927 23:03:59.467729 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 23:03:59 dist-test-slave-lmfp krb5kdc[5727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260927 23:03:59.505191 5978 heartbeater.cc:503] Master 127.3.240.254:45487 was elected leader, sending a full tablet report...
Sep 27 23:03:59 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:59.510717 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.497883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46822 (local address 127.3.240.254:45487)
0927 23:03:59.498075 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.498081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.498344 (+ 263us) server_negotiation.cc:407] Connection header received
0927 23:03:59.498572 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.498578 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.498653 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.498756 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:59.500083 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.500775 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.502098 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.502569 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.502696 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.506291 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:59.506297 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:59.506302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:59.506331 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:59.508681 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:59.509196 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:59.509200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:59.509202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:59.509324 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:59.509603 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:59.509606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:59.509608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:59.509835 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:59.509999 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.510246 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.510388 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:03:59.514201 5771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:03:59.520470 5771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:03:59.542222 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.531790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.3.240.195:45645)
0927 23:03:59.532596 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.532604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.533933 (+ 1329us) server_negotiation.cc:407] Connection header received
0927 23:03:59.534044 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.534051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.534282 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.534505 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:59.535934 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.537159 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.540100 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.540995 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.541133 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.541374 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.541692 (+ 318us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.541861 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":434,"threads_started":1}
I20260927 23:03:59.542251 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.531321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:45645 (local address 127.0.0.1:41704)
0927 23:03:59.533660 (+ 2339us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.533679 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.533804 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.534809 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.534816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.534847 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:59.535701 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.535715 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.537331 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.537336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.539974 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.539994 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.541193 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.541199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.541432 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.541478 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.541637 (+ 159us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.541818 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2202,"mutex_wait_us":464,"thread_start_us":1235,"threads_started":1}
I20260927 23:03:59.543071 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.531084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.3.240.193:37123)
0927 23:03:59.531655 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.531667 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.533202 (+ 1535us) server_negotiation.cc:407] Connection header received
0927 23:03:59.533679 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.533687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.533955 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.534165 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:59.535828 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.537326 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.539458 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.541736 (+ 2278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.541834 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.542022 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.542607 (+ 585us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.542780 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":969,"threads_started":1}
I20260927 23:03:59.544348 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.534021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59230 (local address 127.3.240.194:46785)
0927 23:03:59.535298 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.535309 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.536214 (+ 905us) server_negotiation.cc:407] Connection header received
0927 23:03:59.536309 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.536315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.536503 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.536669 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:59.537335 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.538744 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.541579 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.542836 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.542983 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.543181 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.544060 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.544169 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1164,"thread_start_us":1048,"threads_started":1}
I20260927 23:03:59.544605 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.533769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46785 (local address 127.0.0.1:59230)
0927 23:03:59.535970 (+ 2201us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.535990 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.536097 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.536842 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.536846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.536867 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:59.537169 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.537178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.538866 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.538871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.541414 (+ 2543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.541428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.543036 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.543042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.543406 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.543437 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.543530 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.543642 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2086,"mutex_wait_us":29,"thread_start_us":1812,"threads_started":1}
I20260927 23:03:59.545680 6186 tablet_service.cc:1467] Processing CreateTablet for tablet a86291c4102642baa632889b8968c185 (DEFAULT_TABLE table=test-table [id=65c222fab92f4540ae547b2526519244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:59.546746 6186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86291c4102642baa632889b8968c185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:59.547116 6047 tablet_service.cc:1467] Processing CreateTablet for tablet a86291c4102642baa632889b8968c185 (DEFAULT_TABLE table=test-table [id=65c222fab92f4540ae547b2526519244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:59.547663 6271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.530842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37123 (local address 127.0.0.1:57666)
0927 23:03:59.533029 (+ 2187us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.533071 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.533388 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.534490 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.534510 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.534547 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:59.535603 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.535632 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.537483 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.537489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.539265 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.539277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.542222 (+ 2945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.542227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.542391 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.542434 (+ 43us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.543302 (+ 868us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.547326 (+ 4024us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1884,"mutex_wait_us":3899,"thread_start_us":2434,"threads_started":1}
I20260927 23:03:59.548235 6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86291c4102642baa632889b8968c185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:59.550730 5908 tablet_service.cc:1467] Processing CreateTablet for tablet a86291c4102642baa632889b8968c185 (DEFAULT_TABLE table=test-table [id=65c222fab92f4540ae547b2526519244]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:03:59.551872 5908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a86291c4102642baa632889b8968c185. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:59.557683 6275 tablet_bootstrap.cc:492] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Bootstrap starting.
I20260927 23:03:59.560607 6275 tablet_bootstrap.cc:654] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:59.561513 6276 tablet_bootstrap.cc:492] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Bootstrap starting.
I20260927 23:03:59.562137 6275 log.cc:824] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:59.564888 6278 tablet_bootstrap.cc:492] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Bootstrap starting.
I20260927 23:03:59.565232 6276 tablet_bootstrap.cc:654] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:59.565064 6275 tablet_bootstrap.cc:492] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: No bootstrap required, opened a new log
I20260927 23:03:59.565652 6275 ts_tablet_manager.cc:1397] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260927 23:03:59.566481 6276 log.cc:824] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:59.567852 6278 tablet_bootstrap.cc:654] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:59.568718 6278 log.cc:824] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Log is configured to *not* fsync() on all Append() calls
I20260927 23:03:59.569254 6276 tablet_bootstrap.cc:492] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: No bootstrap required, opened a new log
I20260927 23:03:59.569495 6276 ts_tablet_manager.cc:1397] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 23:03:59.571319 6278 tablet_bootstrap.cc:492] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: No bootstrap required, opened a new log
I20260927 23:03:59.571632 6278 ts_tablet_manager.cc:1397] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:03:59.575409 6275 raft_consensus.cc:348] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.575806 6275 raft_consensus.cc:374] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:59.575937 6275 raft_consensus.cc:729] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddd2fd7ccb14ddebc16f3b2c1a3dfaf, State: Initialized, Role: FOLLOWER
I20260927 23:03:59.576499 6275 consensus_queue.cc:260] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.577569 6256 heartbeater.cc:503] Master 127.3.240.254:45487 was elected leader, sending a full tablet report...
I20260927 23:03:59.578176 6275 ts_tablet_manager.cc:1428] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260927 23:03:59.578744 6278 raft_consensus.cc:348] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.579175 6278 raft_consensus.cc:374] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:59.579286 6278 raft_consensus.cc:729] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c725c7ecf4944cab607f1eb06de1772, State: Initialized, Role: FOLLOWER
I20260927 23:03:59.579795 6278 consensus_queue.cc:260] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.579767 6276 raft_consensus.cc:348] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.580134 6276 raft_consensus.cc:374] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:59.580291 6276 raft_consensus.cc:729] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5ac701368640d7a33a2a9c550872f8, State: Initialized, Role: FOLLOWER
I20260927 23:03:59.580942 6276 consensus_queue.cc:260] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.581270 6278 ts_tablet_manager.cc:1428] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 23:03:59.583182 6117 heartbeater.cc:503] Master 127.3.240.254:45487 was elected leader, sending a full tablet report...
I20260927 23:03:59.584764 6276 ts_tablet_manager.cc:1428] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260927 23:03:59.649879 6281 raft_consensus.cc:482] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:03:59.650131 6281 raft_consensus.cc:504] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.651875 6281 leader_election.cc:288] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123), fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785)
I20260927 23:03:59.657451 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.652077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37123 (local address 127.3.240.195:60513)
0927 23:03:59.652270 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.652289 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.652411 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.652784 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.652790 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.652814 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:59.653022 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.653037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.654251 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.654257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.655901 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.655916 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.656948 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.656952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.657088 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.657110 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.657178 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.657261 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":24}
I20260927 23:03:59.657567 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.652290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:60513 (local address 127.3.240.193:37123)
0927 23:03:59.652468 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.652475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.652493 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:03:59.652571 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.652575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.652642 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.652727 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:59.653199 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.654117 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.656062 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.656802 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.656970 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.657040 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.657220 (+ 180us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.657342 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:03:59.658635 6284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.652408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46785 (local address 127.3.240.195:32805)
0927 23:03:59.653343 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.653357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.653490 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.653835 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.653840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.653863 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:59.654120 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.654127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.655161 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.655166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.656761 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.656771 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.658124 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.658128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.658231 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.658252 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.658319 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.658393 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"mutex_wait_us":3,"thread_start_us":673,"threads_started":1}
I20260927 23:03:59.658701 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.652647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:32805 (local address 127.3.240.194:46785)
0927 23:03:59.652871 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.652880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.653439 (+ 559us) server_negotiation.cc:407] Connection header received
0927 23:03:59.653642 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.653648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.653713 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.653794 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:59.654286 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.654943 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.656900 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.657942 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.658107 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.658180 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.658357 (+ 177us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.658476 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 23:03:59.658535 5933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86291c4102642baa632889b8968c185" candidate_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c725c7ecf4944cab607f1eb06de1772" is_pre_election: true
I20260927 23:03:59.659085 5933 raft_consensus.cc:2454] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fddd2fd7ccb14ddebc16f3b2c1a3dfaf in term 0.
I20260927 23:03:59.659600 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86291c4102642baa632889b8968c185" candidate_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5ac701368640d7a33a2a9c550872f8" is_pre_election: true
I20260927 23:03:59.659739 6139 leader_election.cc:302] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fddd2fd7ccb14ddebc16f3b2c1a3dfaf; no voters:
I20260927 23:03:59.660115 6281 raft_consensus.cc:2781] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:03:59.660092 6071 raft_consensus.cc:2454] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fddd2fd7ccb14ddebc16f3b2c1a3dfaf in term 0.
I20260927 23:03:59.660247 6281 raft_consensus.cc:482] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:03:59.660357 6281 raft_consensus.cc:3037] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:59.662635 6281 raft_consensus.cc:504] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.663187 6281 leader_election.cc:288] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [CANDIDATE]: Term 1 election: Requested vote from peers 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123), fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785)
I20260927 23:03:59.663578 5933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86291c4102642baa632889b8968c185" candidate_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c725c7ecf4944cab607f1eb06de1772"
I20260927 23:03:59.663645 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a86291c4102642baa632889b8968c185" candidate_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5ac701368640d7a33a2a9c550872f8"
I20260927 23:03:59.663774 5933 raft_consensus.cc:3037] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:59.663806 6071 raft_consensus.cc:3037] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:59.666873 6071 raft_consensus.cc:2454] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fddd2fd7ccb14ddebc16f3b2c1a3dfaf in term 1.
I20260927 23:03:59.666867 5933 raft_consensus.cc:2454] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fddd2fd7ccb14ddebc16f3b2c1a3dfaf in term 1.
I20260927 23:03:59.667219 6139 leader_election.cc:302] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fddd2fd7ccb14ddebc16f3b2c1a3dfaf; no voters:
I20260927 23:03:59.667454 6281 raft_consensus.cc:2781] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:59.667886 6281 raft_consensus.cc:686] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 LEADER]: Becoming Leader. State: Replica: fddd2fd7ccb14ddebc16f3b2c1a3dfaf, State: Running, Role: LEADER
I20260927 23:03:59.668365 6281 consensus_queue.cc:237] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.672683 5772 catalog_manager.cc:5495] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf reported cstate change: term changed from 0 to 1, leader changed from <none> to fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195). New cstate: current_term: 1 leader_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } health_report { overall_health: HEALTHY } } }
W20260927 23:03:59.710227 6257 tablet.cc:2369] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:03:59.716501 6118 tablet.cc:2369] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 27 23:03:59 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550237, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:03:59.733239 5820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.717632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46826 (local address 127.3.240.254:45487)
0927 23:03:59.717809 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.717817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.718407 (+ 590us) server_negotiation.cc:407] Connection header received
0927 23:03:59.718596 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.718601 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.718686 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.718788 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:03:59.720341 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.721458 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.723304 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.723716 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.723844 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.727875 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:03:59.727881 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:03:59.727885 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:03:59.727917 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:03:59.730375 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:59.731204 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:59.731208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:59.731210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:59.731347 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:03:59.731733 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:03:59.731738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:03:59.731745 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:03:59.731995 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:03:59.732194 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.732583 (+ 389us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.732894 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":55}
I20260927 23:03:59.733260 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.717714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45487 (local address 127.0.0.1:46826)
0927 23:03:59.718290 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.718308 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.718451 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.718851 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.718858 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.719800 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:03:59.720188 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.720202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.721657 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.721664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.723188 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.723201 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.723878 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.723884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.724039 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.724770 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:03:59.724791 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:03:59.727677 (+ 2886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:03:59.730514 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:59.730524 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:59.730530 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:59.730935 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:59.731477 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:03:59.731484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:03:59.731487 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:03:59.731626 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:03:59.732176 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:03:59.732183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:03:59.732341 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.732496 (+ 155us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.732764 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":310,"threads_started":1}
I20260927 23:03:59.739189 5772 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:46826:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 23:03:59.740569 5772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:03:59.746672 5908 tablet_service.cc:1467] Processing CreateTablet for tablet 17ea1f2dd49f44e59eac0fffea27add9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72f5a2b936a844e290e9b4976eb5a9ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:03:59.746829 6047 tablet_service.cc:1467] Processing CreateTablet for tablet 17ea1f2dd49f44e59eac0fffea27add9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72f5a2b936a844e290e9b4976eb5a9ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:03:59.747617 5908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ea1f2dd49f44e59eac0fffea27add9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:59.747679 6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ea1f2dd49f44e59eac0fffea27add9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 23:03:59.748883 5979 tablet.cc:2369] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:03:59.751506 6186 tablet_service.cc:1467] Processing CreateTablet for tablet 17ea1f2dd49f44e59eac0fffea27add9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72f5a2b936a844e290e9b4976eb5a9ab]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:03:59.752314 6186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17ea1f2dd49f44e59eac0fffea27add9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:03:59.752982 6278 tablet_bootstrap.cc:492] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: Bootstrap starting.
I20260927 23:03:59.753126 6276 tablet_bootstrap.cc:492] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: Bootstrap starting.
I20260927 23:03:59.754096 6278 tablet_bootstrap.cc:654] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:59.754659 6276 tablet_bootstrap.cc:654] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:59.756398 6278 tablet_bootstrap.cc:492] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: No bootstrap required, opened a new log
I20260927 23:03:59.756552 6278 ts_tablet_manager.cc:1397] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260927 23:03:59.757009 6276 tablet_bootstrap.cc:492] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: No bootstrap required, opened a new log
I20260927 23:03:59.757130 6276 ts_tablet_manager.cc:1397] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 23:03:59.757143 6278 raft_consensus.cc:348] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.757361 6278 raft_consensus.cc:374] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:59.757444 6278 raft_consensus.cc:729] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c725c7ecf4944cab607f1eb06de1772, State: Initialized, Role: FOLLOWER
I20260927 23:03:59.757645 6278 consensus_queue.cc:260] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.757783 6276 raft_consensus.cc:348] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.758059 6276 raft_consensus.cc:374] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:59.758139 6276 raft_consensus.cc:729] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5ac701368640d7a33a2a9c550872f8, State: Initialized, Role: FOLLOWER
I20260927 23:03:59.758287 6278 ts_tablet_manager.cc:1428] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 23:03:59.758127 6282 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.758361 6276 consensus_queue.cc:260] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.758512 6282 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:59.758920 6276 ts_tablet_manager.cc:1428] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 23:03:59.758769 6283 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.759606 6283 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:59.761574 6275 tablet_bootstrap.cc:492] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Bootstrap starting.
I20260927 23:03:59.762807 6275 tablet_bootstrap.cc:654] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Neither blocks nor log segments found. Creating new log.
I20260927 23:03:59.764636 6275 tablet_bootstrap.cc:492] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: No bootstrap required, opened a new log
I20260927 23:03:59.764746 6275 ts_tablet_manager.cc:1397] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 23:03:59.765219 6275 raft_consensus.cc:348] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.765367 6275 raft_consensus.cc:374] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:03:59.765448 6275 raft_consensus.cc:729] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddd2fd7ccb14ddebc16f3b2c1a3dfaf, State: Initialized, Role: FOLLOWER
I20260927 23:03:59.765645 6275 consensus_queue.cc:260] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.766155 6275 ts_tablet_manager.cc:1428] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260927 23:03:59.765996 6281 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.766322 6281 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:59.858857 6283 raft_consensus.cc:482] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:03:59.859130 6283 raft_consensus.cc:504] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.860710 6283 leader_election.cc:288] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123), fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:03:59.867087 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.861072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:43171 (local address 127.3.240.195:45645)
0927 23:03:59.861282 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.861290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.862029 (+ 739us) server_negotiation.cc:407] Connection header received
0927 23:03:59.862183 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.862187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.862235 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.862321 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:59.862973 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.863674 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.865424 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.866133 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.866284 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.866334 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.866675 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.866797 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:03:59.867134 6270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.860974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:38557 (local address 127.3.240.193:37123)
0927 23:03:59.861266 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.861275 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.861900 (+ 625us) server_negotiation.cc:407] Connection header received
0927 23:03:59.862148 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.862155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.862219 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.862279 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:03:59.863082 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.863799 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.865507 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.866275 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.866434 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.866501 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.866741 (+ 240us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.866866 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 23:03:59.867218 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.860893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37123 (local address 127.3.240.194:38557)
0927 23:03:59.861793 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.861820 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.861970 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.862322 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.862344 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.862365 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:59.862872 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.862882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.863918 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.863921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.865365 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.865381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.866467 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.866471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.866584 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.866604 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.866704 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.866810 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":19,"thread_start_us":697,"threads_started":1}
I20260927 23:03:59.867267 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.861025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:45645 (local address 127.3.240.194:43171)
0927 23:03:59.861925 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.861938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.862067 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.862374 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.862378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.862401 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:03:59.862813 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.862822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.863801 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.863806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.865289 (+ 1483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.865299 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.866333 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.866340 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.866531 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.866562 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.866630 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.866749 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":811,"mutex_wait_us":55,"thread_start_us":457,"threads_started":1}
I20260927 23:03:59.867947 5933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ea1f2dd49f44e59eac0fffea27add9" candidate_uuid: "fd5ac701368640d7a33a2a9c550872f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c725c7ecf4944cab607f1eb06de1772" is_pre_election: true
I20260927 23:03:59.867964 6206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ea1f2dd49f44e59eac0fffea27add9" candidate_uuid: "fd5ac701368640d7a33a2a9c550872f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" is_pre_election: true
I20260927 23:03:59.868191 5933 raft_consensus.cc:2454] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd5ac701368640d7a33a2a9c550872f8 in term 0.
I20260927 23:03:59.868300 6206 raft_consensus.cc:2454] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd5ac701368640d7a33a2a9c550872f8 in term 0.
I20260927 23:03:59.868599 6000 leader_election.cc:302] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fd5ac701368640d7a33a2a9c550872f8; no voters:
I20260927 23:03:59.868934 6283 raft_consensus.cc:2781] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:03:59.869046 6283 raft_consensus.cc:482] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:03:59.869138 6283 raft_consensus.cc:3037] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:59.871300 6283 raft_consensus.cc:504] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.871814 6283 leader_election.cc:288] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 election: Requested vote from peers 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123), fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:03:59.872144 5933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ea1f2dd49f44e59eac0fffea27add9" candidate_uuid: "fd5ac701368640d7a33a2a9c550872f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c725c7ecf4944cab607f1eb06de1772"
I20260927 23:03:59.872224 6206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17ea1f2dd49f44e59eac0fffea27add9" candidate_uuid: "fd5ac701368640d7a33a2a9c550872f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf"
I20260927 23:03:59.872334 5933 raft_consensus.cc:3037] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:59.872416 6206 raft_consensus.cc:3037] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:03:59.874372 6206 raft_consensus.cc:2454] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd5ac701368640d7a33a2a9c550872f8 in term 1.
I20260927 23:03:59.874428 5933 raft_consensus.cc:2454] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd5ac701368640d7a33a2a9c550872f8 in term 1.
I20260927 23:03:59.874719 5998 leader_election.cc:302] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fd5ac701368640d7a33a2a9c550872f8, fddd2fd7ccb14ddebc16f3b2c1a3dfaf; no voters:
I20260927 23:03:59.874972 6283 raft_consensus.cc:2781] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:03:59.875409 6283 raft_consensus.cc:686] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [term 1 LEADER]: Becoming Leader. State: Replica: fd5ac701368640d7a33a2a9c550872f8, State: Running, Role: LEADER
I20260927 23:03:59.875840 6283 consensus_queue.cc:237] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:03:59.879453 6298 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: TxnStatusTablet state changed. Reason: New leader fd5ac701368640d7a33a2a9c550872f8. Latest consensus state: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.879835 6298 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:03:59.880501 6300 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 23:03:59.880731 6300 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 23:03:59.880931 5771 catalog_manager.cc:5495] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } health_report { overall_health: UNKNOWN } } }
I20260927 23:03:59.920104 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.914230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.3.240.194:46785)
0927 23:03:59.914429 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.914435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.914458 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:03:59.914678 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.914685 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.914798 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.914953 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:03:59.915273 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.916388 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.917567 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.918201 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.918313 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.918813 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:03:59.919474 (+ 661us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:03:59.919623 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.919713 (+ 90us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.919788 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:03:59.920127 6292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.914126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46785 (local address 127.0.0.1:59238)
0927 23:03:59.914374 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.914398 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.914503 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.914958 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.914960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.914970 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:03:59.915152 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.915159 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.916537 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.916542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.917435 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.917446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.918357 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.918362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.918587 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.918645 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:03:59.919582 (+ 937us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:03:59.919600 (+ 18us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.919679 (+ 79us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.919827 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 23:03:59.928599 6206 raft_consensus.cc:1261] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Refusing update from remote peer fd5ac701368640d7a33a2a9c550872f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:03:59.928709 5933 raft_consensus.cc:1261] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Refusing update from remote peer fd5ac701368640d7a33a2a9c550872f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:03:59.929508 6298 consensus_queue.cc:1035] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:03:59.929804 6283 consensus_queue.cc:1035] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:03:59.944893 6282 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: TxnStatusTablet state changed. Reason: New leader fd5ac701368640d7a33a2a9c550872f8. Latest consensus state: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.945195 6282 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:59.947046 6281 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: TxnStatusTablet state changed. Reason: New leader fd5ac701368640d7a33a2a9c550872f8. Latest consensus state: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.947243 6303 mvcc.cc:205] Tried to move back new op lower bound from 7334093782738182144 to 7334093782534914048. Current Snapshot: MvccSnapshot[applied={T|T < 7334093782738182144}]
I20260927 23:03:59.947386 6281 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:59.948228 6305 mvcc.cc:205] Tried to move back new op lower bound from 7334093782738182144 to 7334093782534914048. Current Snapshot: MvccSnapshot[applied={T|T < 7334093782738182144}]
I20260927 23:03:59.951258 6283 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.951553 6283 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:03:59.951779 6283 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.952020 6283 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P fd5ac701368640d7a33a2a9c550872f8: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:03:59.954209 6304 mvcc.cc:205] Tried to move back new op lower bound from 7334093782738182144 to 7334093782534914048. Current Snapshot: MvccSnapshot[applied={T|T < 7334093782738182144}]
I20260927 23:03:59.954314 6282 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.954540 6282 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P 0c725c7ecf4944cab607f1eb06de1772: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:59.957536 6281 tablet_replica.cc:439] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } } }
I20260927 23:03:59.957844 6281 tablet_replica.cc:442] T 17ea1f2dd49f44e59eac0fffea27add9 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:03:59.971586 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.964273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.3.240.195:45645)
0927 23:03:59.964449 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.964457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.964689 (+ 232us) server_negotiation.cc:407] Connection header received
0927 23:03:59.964928 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.964934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.965065 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.965215 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:03:59.965726 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.966817 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.968781 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.969407 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.969552 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.970096 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:03:59.970738 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:03:59.970866 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.971069 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.971149 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:03:59.990736 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.983111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:34449 (local address 127.3.240.194:46785)
0927 23:03:59.983304 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 23:03:59.983310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:03:59.986135 (+ 2825us) server_negotiation.cc:407] Connection header received
0927 23:03:59.986326 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:03:59.986332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:03:59.986408 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:03:59.986526 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:03:59.986911 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.987757 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.989089 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.989524 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.989665 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.990070 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:03:59.990159 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:03:59.990250 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 23:03:59.990381 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 23:03:59.990458 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 23:03:59.990945 6315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:03:59.983027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46785 (local address 127.3.240.195:34449)
0927 23:03:59.986037 (+ 3010us) negotiation.cc:107] Waiting for socket to connect
0927 23:03:59.986051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:03:59.986197 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:03:59.986536 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:03:59.986541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:03:59.986551 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:03:59.986779 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.986787 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.987871 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.987877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.988938 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:03:59.988948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:03:59.989698 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:03:59.989704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:03:59.989837 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:03:59.989882 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:03:59.990251 (+ 369us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:03:59.990271 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 23:03:59.990361 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 23:03:59.990496 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2886,"thread_start_us":2757,"threads_started":1}
I20260927 23:03:59.999902 5933 raft_consensus.cc:1261] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Refusing update from remote peer fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:00.000043 6071 raft_consensus.cc:1261] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Refusing update from remote peer fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:00.000738 6285 consensus_queue.cc:1035] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:00.001307 6281 consensus_queue.cc:1035] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:00.039939 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:00.033970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:56781 (local address 127.3.240.195:45645)
0927 23:04:00.034138 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:00.034155 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:00.034842 (+ 687us) server_negotiation.cc:407] Connection header received
0927 23:04:00.035053 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:00.035057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:00.035120 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:00.035199 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:00.035717 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.036585 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.037961 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.038544 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.038666 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:00.039216 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:00.039339 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:00.039444 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:00.039565 (+ 121us) server_negotiation.cc:299] Negotiation successful
0927 23:04:00.039652 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:04:00.040120 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:00.034066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:45645 (local address 127.3.240.194:56781)
0927 23:04:00.034745 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:00.034772 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:00.034903 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:00.035213 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:00.035217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:00.035226 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:04:00.035572 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:00.035579 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.036694 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.036701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:00.037801 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:00.037815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.038710 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.038717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:00.038952 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:00.039043 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:04:00.039444 (+ 401us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:04:00.039457 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 23:04:00.039542 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 23:04:00.039688 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":376,"threads_started":1}
I20260927 23:04:00.080894 5772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:00.081176 5772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:00.084980 5772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: Sending DeleteTablet for 3 replicas of tablet a86291c4102642baa632889b8968c185
I20260927 23:04:00.086056 6047 tablet_service.cc:1514] Processing DeleteTablet for tablet a86291c4102642baa632889b8968c185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:00 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:59230
I20260927 23:04:00.086076 5908 tablet_service.cc:1514] Processing DeleteTablet for tablet a86291c4102642baa632889b8968c185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:00 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57666
I20260927 23:04:00.087035 6186 tablet_service.cc:1514] Processing DeleteTablet for tablet a86291c4102642baa632889b8968c185 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:00 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41704
I20260927 23:04:00.093567 6325 tablet_replica.cc:330] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: stopping tablet replica
I20260927 23:04:00.093508 5772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 23:04:00.094017 6325 raft_consensus.cc:2229] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 LEADER]: Raft consensus shutting down.
W20260927 23:04:00.094105 5772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:00.094681 6325 raft_consensus.cc:2258] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:00.101204 5907 tablet_service.cc:1467] Processing CreateTablet for tablet 084424b39cc84e4cab733f32e82c1214 (DEFAULT_TABLE table=test-table [id=f303b5db67ac4882b37d7e8520c2d32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:00.101356 6046 tablet_service.cc:1467] Processing CreateTablet for tablet 084424b39cc84e4cab733f32e82c1214 (DEFAULT_TABLE table=test-table [id=f303b5db67ac4882b37d7e8520c2d32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:00.101749 5907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084424b39cc84e4cab733f32e82c1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:00.101843 6046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084424b39cc84e4cab733f32e82c1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:00.107379 6186 tablet_service.cc:1467] Processing CreateTablet for tablet 084424b39cc84e4cab733f32e82c1214 (DEFAULT_TABLE table=test-table [id=f303b5db67ac4882b37d7e8520c2d32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:00.107761 6186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 084424b39cc84e4cab733f32e82c1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:00.108791 6278 tablet_bootstrap.cc:492] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: Bootstrap starting.
I20260927 23:04:00.110327 6278 tablet_bootstrap.cc:654] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:00.110522 6325 ts_tablet_manager.cc:1905] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:00.117916 6325 ts_tablet_manager.cc:1918] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:00.120906 6276 tablet_bootstrap.cc:492] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: Bootstrap starting.
I20260927 23:04:00.121937 6324 tablet_replica.cc:330] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: stopping tablet replica
I20260927 23:04:00.122522 6324 raft_consensus.cc:2229] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:00.122630 6276 tablet_bootstrap.cc:654] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:00.122974 6324 raft_consensus.cc:2258] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:00.123417 6325 log.cc:1196] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/a86291c4102642baa632889b8968c185
I20260927 23:04:00.124058 6325 ts_tablet_manager.cc:1939] T a86291c4102642baa632889b8968c185 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting consensus metadata
I20260927 23:04:00.125523 6324 ts_tablet_manager.cc:1905] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:00.128492 6276 tablet_bootstrap.cc:492] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: No bootstrap required, opened a new log
I20260927 23:04:00.128660 6276 ts_tablet_manager.cc:1397] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260927 23:04:00.128954 5756 catalog_manager.cc:4841] TS fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645): tablet a86291c4102642baa632889b8968c185 (table test-table [id=65c222fab92f4540ae547b2526519244]) successfully deleted
I20260927 23:04:00.129418 6276 raft_consensus.cc:348] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.129667 6276 raft_consensus.cc:374] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:00.129750 6276 raft_consensus.cc:729] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5ac701368640d7a33a2a9c550872f8, State: Initialized, Role: FOLLOWER
I20260927 23:04:00.130033 6276 consensus_queue.cc:260] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.131527 6276 ts_tablet_manager.cc:1428] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260927 23:04:00.135687 6324 ts_tablet_manager.cc:1918] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:00.135885 6324 log.cc:1196] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/a86291c4102642baa632889b8968c185
I20260927 23:04:00.136423 6324 ts_tablet_manager.cc:1939] T a86291c4102642baa632889b8968c185 P 0c725c7ecf4944cab607f1eb06de1772: Deleting consensus metadata
I20260927 23:04:00.138181 5758 catalog_manager.cc:4841] TS 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123): tablet a86291c4102642baa632889b8968c185 (table test-table [id=65c222fab92f4540ae547b2526519244]) successfully deleted
I20260927 23:04:00.143316 6275 tablet_bootstrap.cc:492] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Bootstrap starting.
I20260927 23:04:00.144472 6323 tablet_replica.cc:330] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: stopping tablet replica
I20260927 23:04:00.144997 6275 tablet_bootstrap.cc:654] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:00.144996 6323 raft_consensus.cc:2229] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:00.145566 6323 raft_consensus.cc:2258] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:00.148118 6278 tablet_bootstrap.cc:492] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: No bootstrap required, opened a new log
I20260927 23:04:00.148343 6278 ts_tablet_manager.cc:1397] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.005s
I20260927 23:04:00.148367 6298 raft_consensus.cc:482] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:00.148519 6298 raft_consensus.cc:504] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.149091 6298 leader_election.cc:288] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123), fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:04:00.149313 6278 raft_consensus.cc:348] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.149677 6205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084424b39cc84e4cab733f32e82c1214" candidate_uuid: "fd5ac701368640d7a33a2a9c550872f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" is_pre_election: true
I20260927 23:04:00.149808 6278 raft_consensus.cc:374] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:00.150115 6278 raft_consensus.cc:729] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c725c7ecf4944cab607f1eb06de1772, State: Initialized, Role: FOLLOWER
W20260927 23:04:00.150408 5998 leader_election.cc:341] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645): Illegal state: must be running to vote when last-logged opid is not known
I20260927 23:04:00.150425 6278 consensus_queue.cc:260] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.151196 5933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084424b39cc84e4cab733f32e82c1214" candidate_uuid: "fd5ac701368640d7a33a2a9c550872f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c725c7ecf4944cab607f1eb06de1772" is_pre_election: true
I20260927 23:04:00.151381 5933 raft_consensus.cc:2454] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd5ac701368640d7a33a2a9c550872f8 in term 0.
I20260927 23:04:00.153033 6323 ts_tablet_manager.cc:1905] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:00.153733 6000 leader_election.cc:302] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fd5ac701368640d7a33a2a9c550872f8; no voters: fddd2fd7ccb14ddebc16f3b2c1a3dfaf
I20260927 23:04:00.154827 6298 raft_consensus.cc:2781] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:00.154944 6298 raft_consensus.cc:482] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:00.151203 6278 ts_tablet_manager.cc:1428] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 23:04:00.155644 6298 raft_consensus.cc:3037] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:00.158083 6275 tablet_bootstrap.cc:492] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: No bootstrap required, opened a new log
I20260927 23:04:00.158314 6275 ts_tablet_manager.cc:1397] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20260927 23:04:00.158885 6298 raft_consensus.cc:504] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.159447 6275 raft_consensus.cc:348] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.159742 6275 raft_consensus.cc:374] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:00.159870 6275 raft_consensus.cc:729] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddd2fd7ccb14ddebc16f3b2c1a3dfaf, State: Initialized, Role: FOLLOWER
I20260927 23:04:00.159943 5933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084424b39cc84e4cab733f32e82c1214" candidate_uuid: "fd5ac701368640d7a33a2a9c550872f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c725c7ecf4944cab607f1eb06de1772"
I20260927 23:04:00.160123 5933 raft_consensus.cc:3037] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:00.160337 6275 consensus_queue.cc:260] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.162928 5933 raft_consensus.cc:2454] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd5ac701368640d7a33a2a9c550872f8 in term 1.
I20260927 23:04:00.163120 6323 ts_tablet_manager.cc:1918] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:00.163290 6323 log.cc:1196] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/a86291c4102642baa632889b8968c185
I20260927 23:04:00.163410 6000 leader_election.cc:302] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fd5ac701368640d7a33a2a9c550872f8; no voters:
I20260927 23:04:00.163715 6310 raft_consensus.cc:2781] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:00.163867 6310 raft_consensus.cc:686] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 1 LEADER]: Becoming Leader. State: Replica: fd5ac701368640d7a33a2a9c550872f8, State: Running, Role: LEADER
I20260927 23:04:00.164283 6275 ts_tablet_manager.cc:1428] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260927 23:04:00.164188 6310 consensus_queue.cc:237] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:00.164518 6298 leader_election.cc:288] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [CANDIDATE]: Term 1 election: Requested vote from peers 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123), fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:04:00.165593 6205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "084424b39cc84e4cab733f32e82c1214" candidate_uuid: "fd5ac701368640d7a33a2a9c550872f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf"
I20260927 23:04:00.165792 6205 raft_consensus.cc:3037] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:00.168876 6205 raft_consensus.cc:2454] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd5ac701368640d7a33a2a9c550872f8 in term 1.
I20260927 23:04:00.163851 6323 ts_tablet_manager.cc:1939] T a86291c4102642baa632889b8968c185 P fd5ac701368640d7a33a2a9c550872f8: Deleting consensus metadata
I20260927 23:04:00.173619 5758 catalog_manager.cc:4841] TS fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785): tablet a86291c4102642baa632889b8968c185 (table test-table [id=65c222fab92f4540ae547b2526519244]) successfully deleted
I20260927 23:04:00.178777 5772 catalog_manager.cc:5495] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "fd5ac701368640d7a33a2a9c550872f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } health_report { overall_health: UNKNOWN } } }
I20260927 23:04:00.209198 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:00.202433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59250 (local address 127.3.240.194:46785)
0927 23:04:00.202618 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:00.202624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:00.202642 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:04:00.202711 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:00.202716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:00.202791 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:00.202875 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:00.203467 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.204617 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.206685 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.207369 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.207506 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:00.208046 (+ 540us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:00.208180 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:00.208254 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:00.208783 (+ 529us) server_negotiation.cc:299] Negotiation successful
0927 23:04:00.208860 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 23:04:00.224622 6319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:00.218421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46785 (local address 127.3.240.194:58241)
0927 23:04:00.218603 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:00.218628 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:00.218742 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:00.219110 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:00.219120 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:00.219133 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:04:00.219313 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:00.219324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.220675 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.220680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:00.222242 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:00.222255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.223267 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.223274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:00.223461 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:00.223507 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:04:00.224013 (+ 506us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:04:00.224025 (+ 12us) client_negotiation.cc:769] Sending connection context
0927 23:04:00.224114 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 23:04:00.224253 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 23:04:00.224622 6274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:00.218669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:58241 (local address 127.3.240.194:46785)
0927 23:04:00.218822 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:00.218827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:00.218843 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:04:00.218899 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:00.218904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:00.218974 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:00.219085 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:00.219456 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.220541 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.222380 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:00.223089 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:00.223205 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:00.223688 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:00.223859 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:00.223978 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:00.224147 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 23:04:00.224230 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":43}
I20260927 23:04:00.234545 6206 raft_consensus.cc:1261] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Refusing update from remote peer fd5ac701368640d7a33a2a9c550872f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:00.234731 5933 raft_consensus.cc:1261] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Refusing update from remote peer fd5ac701368640d7a33a2a9c550872f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:00.235321 6283 consensus_queue.cc:1035] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:00.235630 6310 consensus_queue.cc:1035] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 23:04:00.277446 6303 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 23:04:01.871662 5771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:01.871891 5771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:01.875278 5771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: Sending DeleteTablet for 3 replicas of tablet 084424b39cc84e4cab733f32e82c1214
I20260927 23:04:01.875927 5908 tablet_service.cc:1514] Processing DeleteTablet for tablet 084424b39cc84e4cab733f32e82c1214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:01 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57666
I20260927 23:04:01.875975 6047 tablet_service.cc:1514] Processing DeleteTablet for tablet 084424b39cc84e4cab733f32e82c1214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:01 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:59230
I20260927 23:04:01.876189 6186 tablet_service.cc:1514] Processing DeleteTablet for tablet 084424b39cc84e4cab733f32e82c1214 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:01 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41704
I20260927 23:04:01.879385 6356 tablet_replica.cc:330] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: stopping tablet replica
I20260927 23:04:01.879546 6355 tablet_replica.cc:330] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: stopping tablet replica
I20260927 23:04:01.879593 6356 raft_consensus.cc:2229] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:01.879716 6355 raft_consensus.cc:2229] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:01.879810 6356 raft_consensus.cc:2258] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:01.880275 6355 raft_consensus.cc:2258] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:01.880716 6356 ts_tablet_manager.cc:1905] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:01.881089 6355 ts_tablet_manager.cc:1905] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:01.887534 6354 tablet_replica.cc:330] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: stopping tablet replica
I20260927 23:04:01.887615 6356 ts_tablet_manager.cc:1918] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 23:04:01.887717 6356 log.cc:1196] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/084424b39cc84e4cab733f32e82c1214
I20260927 23:04:01.887728 6354 raft_consensus.cc:2229] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:01.887940 6354 raft_consensus.cc:2258] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:01.888392 6355 ts_tablet_manager.cc:1918] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 23:04:01.888578 6356 ts_tablet_manager.cc:1939] T 084424b39cc84e4cab733f32e82c1214 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting consensus metadata
I20260927 23:04:01.888589 6355 log.cc:1196] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/084424b39cc84e4cab733f32e82c1214
I20260927 23:04:01.888842 6354 ts_tablet_manager.cc:1905] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:01.889153 6355 ts_tablet_manager.cc:1939] T 084424b39cc84e4cab733f32e82c1214 P fd5ac701368640d7a33a2a9c550872f8: Deleting consensus metadata
I20260927 23:04:01.889603 5756 catalog_manager.cc:4841] TS fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645): tablet 084424b39cc84e4cab733f32e82c1214 (table test-table [id=f303b5db67ac4882b37d7e8520c2d32f]) successfully deleted
I20260927 23:04:01.890311 5758 catalog_manager.cc:4841] TS fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785): tablet 084424b39cc84e4cab733f32e82c1214 (table test-table [id=f303b5db67ac4882b37d7e8520c2d32f]) successfully deleted
I20260927 23:04:01.893524 6354 ts_tablet_manager.cc:1918] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 23:04:01.893661 6354 log.cc:1196] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/084424b39cc84e4cab733f32e82c1214
I20260927 23:04:01.894052 6354 ts_tablet_manager.cc:1939] T 084424b39cc84e4cab733f32e82c1214 P 0c725c7ecf4944cab607f1eb06de1772: Deleting consensus metadata
I20260927 23:04:01.895077 5758 catalog_manager.cc:4841] TS 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123): tablet 084424b39cc84e4cab733f32e82c1214 (table test-table [id=f303b5db67ac4882b37d7e8520c2d32f]) successfully deleted
Sep 27 23:04:01 dist-test-slave-lmfp krb5kdc[5727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 23:04:01 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:01.930932 6364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:01.913447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.3.240.254:45487)
0927 23:04:01.914004 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:01.914012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:01.914037 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:04:01.914244 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:01.914248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:01.914304 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:01.914408 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:01.915675 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:01.916394 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:01.917438 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:01.917804 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:01.917925 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:01.925820 (+ 7895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:01.925825 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:01.925828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:01.925858 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:01.928674 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:01.929225 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:01.929230 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:01.929233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:01.929340 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:01.929670 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:01.929675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:01.929678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:01.929928 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:01.930157 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:01.930414 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 23:04:01.930612 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":230,"threads_started":1}
I20260927 23:04:01.934082 5771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:01.934628 5771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:01.940419 5908 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1d46a1e3b54be5975fedddcc580391 (DEFAULT_TABLE table=test-table [id=75d822af23fb45349ba590c01df83157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:01.940551 6047 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1d46a1e3b54be5975fedddcc580391 (DEFAULT_TABLE table=test-table [id=75d822af23fb45349ba590c01df83157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:01.940742 6186 tablet_service.cc:1467] Processing CreateTablet for tablet 0a1d46a1e3b54be5975fedddcc580391 (DEFAULT_TABLE table=test-table [id=75d822af23fb45349ba590c01df83157]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:01.940831 5908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1d46a1e3b54be5975fedddcc580391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:01.940928 6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1d46a1e3b54be5975fedddcc580391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:01.941102 6186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a1d46a1e3b54be5975fedddcc580391. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:01.946499 6365 tablet_bootstrap.cc:492] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: Bootstrap starting.
I20260927 23:04:01.947459 6367 tablet_bootstrap.cc:492] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: Bootstrap starting.
I20260927 23:04:01.947760 6366 tablet_bootstrap.cc:492] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Bootstrap starting.
I20260927 23:04:01.948235 6365 tablet_bootstrap.cc:654] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:01.949097 6366 tablet_bootstrap.cc:654] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:01.949121 6367 tablet_bootstrap.cc:654] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:01.951206 6366 tablet_bootstrap.cc:492] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: No bootstrap required, opened a new log
I20260927 23:04:01.951339 6366 ts_tablet_manager.cc:1397] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 23:04:01.951583 6367 tablet_bootstrap.cc:492] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: No bootstrap required, opened a new log
I20260927 23:04:01.951704 6367 ts_tablet_manager.cc:1397] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260927 23:04:01.951936 6366 raft_consensus.cc:348] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:01.952134 6366 raft_consensus.cc:374] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:01.952230 6366 raft_consensus.cc:729] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddd2fd7ccb14ddebc16f3b2c1a3dfaf, State: Initialized, Role: FOLLOWER
I20260927 23:04:01.952294 6367 raft_consensus.cc:348] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:01.952483 6367 raft_consensus.cc:374] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:01.952598 6367 raft_consensus.cc:729] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5ac701368640d7a33a2a9c550872f8, State: Initialized, Role: FOLLOWER
I20260927 23:04:01.952473 6366 consensus_queue.cc:260] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:01.952831 6367 consensus_queue.cc:260] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:01.953401 6367 ts_tablet_manager.cc:1428] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 23:04:01.953431 6366 ts_tablet_manager.cc:1428] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 23:04:01.954301 6365 tablet_bootstrap.cc:492] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: No bootstrap required, opened a new log
I20260927 23:04:01.954396 6365 ts_tablet_manager.cc:1397] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260927 23:04:01.954916 6365 raft_consensus.cc:348] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:01.955168 6365 raft_consensus.cc:374] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:01.955259 6365 raft_consensus.cc:729] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c725c7ecf4944cab607f1eb06de1772, State: Initialized, Role: FOLLOWER
I20260927 23:04:01.955456 6365 consensus_queue.cc:260] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:01.956224 6365 ts_tablet_manager.cc:1428] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 23:04:02.068254 6372 raft_consensus.cc:482] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:02.068449 6372 raft_consensus.cc:504] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.069931 6372 leader_election.cc:288] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785), fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:04:02.087292 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.070389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:47505 (local address 127.3.240.194:46785)
0927 23:04:02.079975 (+ 9586us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:02.079991 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:02.080532 (+ 541us) server_negotiation.cc:407] Connection header received
0927 23:04:02.080603 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:02.080609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:02.080685 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:02.080762 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:02.081610 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.082632 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.085039 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.086214 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.086391 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.086466 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:02.086827 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 23:04:02.086920 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9439,"thread_start_us":9252,"threads_started":1}
I20260927 23:04:02.087373 6373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.070164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46785 (local address 127.3.240.193:47505)
0927 23:04:02.080148 (+ 9984us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:02.080172 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:02.080363 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:02.081094 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:02.081099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:02.081130 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:02.081441 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:02.081455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.082785 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.082788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:02.084892 (+ 2104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:02.084905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.086429 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.086434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:02.086648 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.086694 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 23:04:02.086786 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 23:04:02.086910 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9822,"mutex_wait_us":69,"thread_start_us":9676,"threads_started":1}
I20260927 23:04:02.087536 6374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.070164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:45645 (local address 127.3.240.193:52347)
0927 23:04:02.081163 (+ 10999us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:02.081183 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:02.081316 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:02.081750 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:02.081753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:02.081772 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:02.082014 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:02.082020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.083231 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.083236 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:02.085114 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:02.085123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.086653 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.086658 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:02.086863 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.086890 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 23:04:02.086975 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 23:04:02.087093 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10867,"mutex_wait_us":79,"thread_start_us":10258,"threads_started":1}
I20260927 23:04:02.087724 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.070332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:52347 (local address 127.3.240.195:45645)
0927 23:04:02.080534 (+ 10202us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:02.080541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:02.081280 (+ 739us) server_negotiation.cc:407] Connection header received
0927 23:04:02.081522 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:02.081545 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:02.081620 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:02.081735 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:02.082166 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.083113 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.085230 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.086417 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.086630 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.086677 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:02.087033 (+ 356us) server_negotiation.cc:299] Negotiation successful
0927 23:04:02.087152 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10015,"thread_start_us":9875,"threads_started":1}
I20260927 23:04:02.087961 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1d46a1e3b54be5975fedddcc580391" candidate_uuid: "0c725c7ecf4944cab607f1eb06de1772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5ac701368640d7a33a2a9c550872f8" is_pre_election: true
I20260927 23:04:02.088218 6071 raft_consensus.cc:2454] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c725c7ecf4944cab607f1eb06de1772 in term 0.
I20260927 23:04:02.088315 6206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1d46a1e3b54be5975fedddcc580391" candidate_uuid: "0c725c7ecf4944cab607f1eb06de1772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" is_pre_election: true
I20260927 23:04:02.088562 6206 raft_consensus.cc:2454] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c725c7ecf4944cab607f1eb06de1772 in term 0.
I20260927 23:04:02.088812 5861 leader_election.cc:302] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fd5ac701368640d7a33a2a9c550872f8; no voters:
I20260927 23:04:02.089289 6372 raft_consensus.cc:2781] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:02.089391 6372 raft_consensus.cc:482] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:02.089501 6372 raft_consensus.cc:3037] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:02.091778 6372 raft_consensus.cc:504] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.092414 6372 leader_election.cc:288] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [CANDIDATE]: Term 1 election: Requested vote from peers fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785), fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:04:02.092712 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1d46a1e3b54be5975fedddcc580391" candidate_uuid: "0c725c7ecf4944cab607f1eb06de1772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5ac701368640d7a33a2a9c550872f8"
I20260927 23:04:02.092890 6071 raft_consensus.cc:3037] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:02.092815 6206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a1d46a1e3b54be5975fedddcc580391" candidate_uuid: "0c725c7ecf4944cab607f1eb06de1772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf"
I20260927 23:04:02.093075 6206 raft_consensus.cc:3037] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:02.094836 6071 raft_consensus.cc:2454] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c725c7ecf4944cab607f1eb06de1772 in term 1.
I20260927 23:04:02.094919 6206 raft_consensus.cc:2454] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c725c7ecf4944cab607f1eb06de1772 in term 1.
I20260927 23:04:02.095263 5861 leader_election.cc:302] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fd5ac701368640d7a33a2a9c550872f8; no voters:
I20260927 23:04:02.095527 6372 raft_consensus.cc:2781] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:02.096295 6372 raft_consensus.cc:686] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 LEADER]: Becoming Leader. State: Replica: 0c725c7ecf4944cab607f1eb06de1772, State: Running, Role: LEADER
I20260927 23:04:02.096701 6372 consensus_queue.cc:237] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.099581 5770 catalog_manager.cc:5495] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "0c725c7ecf4944cab607f1eb06de1772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } health_report { overall_health: UNKNOWN } } }
I20260927 23:04:02.146155 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.140379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.3.240.193:37123)
0927 23:04:02.140903 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:02.140907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:02.140923 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:04:02.140999 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:02.141020 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:02.141119 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:02.141298 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:02.141706 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.142486 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.143872 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.144337 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.144490 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.144881 (+ 391us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:02.145558 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:02.145713 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:02.145766 (+ 53us) server_negotiation.cc:299] Negotiation successful
0927 23:04:02.145836 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":309,"threads_started":1}
I20260927 23:04:02.160203 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.155731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:50689 (local address 127.3.240.194:46785)
0927 23:04:02.155878 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:02.155882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:02.156099 (+ 217us) server_negotiation.cc:407] Connection header received
0927 23:04:02.156229 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:02.156233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:02.156287 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:02.156373 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:02.156793 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.157457 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.158785 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.159249 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.159336 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.159704 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:02.159795 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:02.159856 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:02.159964 (+ 108us) server_negotiation.cc:299] Negotiation successful
0927 23:04:02.160028 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 23:04:02.160285 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.155478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46785 (local address 127.3.240.193:50689)
0927 23:04:02.156021 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:02.156036 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:02.156138 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:02.156397 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:02.156400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:02.156408 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:04:02.156657 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:02.156664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.157564 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.157568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:02.158674 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:02.158684 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.159384 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.159388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:02.159522 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.159569 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:04:02.159870 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:04:02.159878 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 23:04:02.159933 (+ 55us) client_negotiation.cc:240] Negotiation successful
0927 23:04:02.160026 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":289,"threads_started":1}
I20260927 23:04:02.168089 6206 raft_consensus.cc:1261] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Refusing update from remote peer 0c725c7ecf4944cab607f1eb06de1772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:02.168314 6071 raft_consensus.cc:1261] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Refusing update from remote peer 0c725c7ecf4944cab607f1eb06de1772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:02.168746 6377 consensus_queue.cc:1035] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:02.169118 6372 consensus_queue.cc:1035] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:02.192592 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.187504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:46119 (local address 127.3.240.193:37123)
0927 23:04:02.187664 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:02.187671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:02.187799 (+ 128us) server_negotiation.cc:407] Connection header received
0927 23:04:02.187986 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:02.187989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:02.188046 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:02.188155 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:02.188588 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.189261 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.190933 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.191419 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.191543 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.191935 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:02.192040 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:02.192125 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:02.192259 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 23:04:02.192329 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 23:04:02.193125 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:02.187186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37123 (local address 127.3.240.194:46119)
0927 23:04:02.187698 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:02.187718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:02.187872 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:02.188172 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:02.188176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:02.188188 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:04:02.188437 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:02.188445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.189388 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.189393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:02.190712 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:02.190722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:02.191594 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:02.191601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:02.191734 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:02.191768 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:04:02.192140 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:04:02.192151 (+ 11us) client_negotiation.cc:769] Sending connection context
0927 23:04:02.192240 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 23:04:02.192365 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":375,"threads_started":1}
I20260927 23:04:02.218561 5770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:02.218806 5770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:02.221689 5770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: Sending DeleteTablet for 3 replicas of tablet 0a1d46a1e3b54be5975fedddcc580391
I20260927 23:04:02.222507 6186 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a1d46a1e3b54be5975fedddcc580391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:02 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41704
I20260927 23:04:02.222522 6047 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a1d46a1e3b54be5975fedddcc580391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:02 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:59230
I20260927 23:04:02.222574 5908 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a1d46a1e3b54be5975fedddcc580391 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:02 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57666
I20260927 23:04:02.222851 6356 tablet_replica.cc:330] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: stopping tablet replica
I20260927 23:04:02.222901 6354 tablet_replica.cc:330] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: stopping tablet replica
I20260927 23:04:02.223062 6356 raft_consensus.cc:2229] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:02.223125 6354 raft_consensus.cc:2229] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:02.223253 6356 raft_consensus.cc:2258] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:02.223520 6355 tablet_replica.cc:330] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: stopping tablet replica
I20260927 23:04:02.223587 6354 raft_consensus.cc:2258] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:02.223701 6355 raft_consensus.cc:2229] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:02.223875 6355 raft_consensus.cc:2258] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:02.224807 6355 ts_tablet_manager.cc:1905] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:02.224902 6356 ts_tablet_manager.cc:1905] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:02.227646 6354 ts_tablet_manager.cc:1905] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:02.230075 6355 ts_tablet_manager.cc:1918] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:02.230213 6355 log.cc:1196] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/0a1d46a1e3b54be5975fedddcc580391
I20260927 23:04:02.230566 6355 ts_tablet_manager.cc:1939] T 0a1d46a1e3b54be5975fedddcc580391 P fd5ac701368640d7a33a2a9c550872f8: Deleting consensus metadata
I20260927 23:04:02.231277 6356 ts_tablet_manager.cc:1918] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:02.231702 5758 catalog_manager.cc:4841] TS fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785): tablet 0a1d46a1e3b54be5975fedddcc580391 (table test-table [id=75d822af23fb45349ba590c01df83157]) successfully deleted
I20260927 23:04:02.231946 6356 log.cc:1196] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/0a1d46a1e3b54be5975fedddcc580391
I20260927 23:04:02.231899 5770 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260927 23:04:02.232340 6356 ts_tablet_manager.cc:1939] T 0a1d46a1e3b54be5975fedddcc580391 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting consensus metadata
W20260927 23:04:02.232527 5770 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:02.233282 5756 catalog_manager.cc:4841] TS fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645): tablet 0a1d46a1e3b54be5975fedddcc580391 (table test-table [id=75d822af23fb45349ba590c01df83157]) successfully deleted
I20260927 23:04:02.235169 6354 ts_tablet_manager.cc:1918] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:02.235334 6354 log.cc:1196] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/0a1d46a1e3b54be5975fedddcc580391
I20260927 23:04:02.235738 6354 ts_tablet_manager.cc:1939] T 0a1d46a1e3b54be5975fedddcc580391 P 0c725c7ecf4944cab607f1eb06de1772: Deleting consensus metadata
I20260927 23:04:02.236625 5758 catalog_manager.cc:4841] TS 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123): tablet 0a1d46a1e3b54be5975fedddcc580391 (table test-table [id=75d822af23fb45349ba590c01df83157]) successfully deleted
I20260927 23:04:02.239512 5908 tablet_service.cc:1467] Processing CreateTablet for tablet 0d134c5a8c8a4d30a237a5c18b215310 (DEFAULT_TABLE table=test-table [id=69976631918242ebb6de4309dc43b632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:02.239948 5908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d134c5a8c8a4d30a237a5c18b215310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:02.239928 6186 tablet_service.cc:1467] Processing CreateTablet for tablet 0d134c5a8c8a4d30a237a5c18b215310 (DEFAULT_TABLE table=test-table [id=69976631918242ebb6de4309dc43b632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:02.240309 6186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d134c5a8c8a4d30a237a5c18b215310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:02.243604 6047 tablet_service.cc:1467] Processing CreateTablet for tablet 0d134c5a8c8a4d30a237a5c18b215310 (DEFAULT_TABLE table=test-table [id=69976631918242ebb6de4309dc43b632]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:02.244033 6047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d134c5a8c8a4d30a237a5c18b215310. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:02.246451 6366 tablet_bootstrap.cc:492] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Bootstrap starting.
I20260927 23:04:02.246573 6365 tablet_bootstrap.cc:492] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: Bootstrap starting.
I20260927 23:04:02.247990 6366 tablet_bootstrap.cc:654] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:02.248173 6365 tablet_bootstrap.cc:654] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:02.250617 6367 tablet_bootstrap.cc:492] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: Bootstrap starting.
I20260927 23:04:02.252104 6366 tablet_bootstrap.cc:492] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: No bootstrap required, opened a new log
I20260927 23:04:02.252252 6366 ts_tablet_manager.cc:1397] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.003s
I20260927 23:04:02.252583 6367 tablet_bootstrap.cc:654] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:02.252988 6366 raft_consensus.cc:348] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.253232 6366 raft_consensus.cc:374] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:02.253301 6366 raft_consensus.cc:729] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fddd2fd7ccb14ddebc16f3b2c1a3dfaf, State: Initialized, Role: FOLLOWER
I20260927 23:04:02.253346 6365 tablet_bootstrap.cc:492] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: No bootstrap required, opened a new log
I20260927 23:04:02.253490 6365 ts_tablet_manager.cc:1397] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260927 23:04:02.253556 6366 consensus_queue.cc:260] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.254062 6366 ts_tablet_manager.cc:1428] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 23:04:02.254225 6365 raft_consensus.cc:348] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.254468 6365 raft_consensus.cc:374] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:02.254554 6365 raft_consensus.cc:729] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c725c7ecf4944cab607f1eb06de1772, State: Initialized, Role: FOLLOWER
I20260927 23:04:02.255190 6365 consensus_queue.cc:260] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.257476 6365 ts_tablet_manager.cc:1428] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260927 23:04:02.259143 6367 tablet_bootstrap.cc:492] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: No bootstrap required, opened a new log
I20260927 23:04:02.259301 6367 ts_tablet_manager.cc:1397] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260927 23:04:02.260062 6367 raft_consensus.cc:348] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.260293 6367 raft_consensus.cc:374] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:02.260370 6367 raft_consensus.cc:729] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd5ac701368640d7a33a2a9c550872f8, State: Initialized, Role: FOLLOWER
I20260927 23:04:02.260613 6367 consensus_queue.cc:260] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.261161 6367 ts_tablet_manager.cc:1428] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 23:04:02.266525 6372 raft_consensus.cc:482] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:02.266685 6372 raft_consensus.cc:504] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.267401 6372 leader_election.cc:288] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785), fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:04:02.267769 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d134c5a8c8a4d30a237a5c18b215310" candidate_uuid: "0c725c7ecf4944cab607f1eb06de1772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5ac701368640d7a33a2a9c550872f8" is_pre_election: true
I20260927 23:04:02.267985 6071 raft_consensus.cc:2454] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c725c7ecf4944cab607f1eb06de1772 in term 0.
I20260927 23:04:02.267931 6206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d134c5a8c8a4d30a237a5c18b215310" candidate_uuid: "0c725c7ecf4944cab607f1eb06de1772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" is_pre_election: true
I20260927 23:04:02.268164 6206 raft_consensus.cc:2454] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c725c7ecf4944cab607f1eb06de1772 in term 0.
I20260927 23:04:02.268364 5861 leader_election.cc:302] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fd5ac701368640d7a33a2a9c550872f8; no voters:
I20260927 23:04:02.268646 6372 raft_consensus.cc:2781] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:02.268795 6372 raft_consensus.cc:482] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:02.268915 6372 raft_consensus.cc:3037] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:02.271297 6372 raft_consensus.cc:504] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.271865 6372 leader_election.cc:288] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [CANDIDATE]: Term 1 election: Requested vote from peers fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785), fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645)
I20260927 23:04:02.272245 6071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d134c5a8c8a4d30a237a5c18b215310" candidate_uuid: "0c725c7ecf4944cab607f1eb06de1772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd5ac701368640d7a33a2a9c550872f8"
I20260927 23:04:02.272280 6206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d134c5a8c8a4d30a237a5c18b215310" candidate_uuid: "0c725c7ecf4944cab607f1eb06de1772" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf"
I20260927 23:04:02.272375 6071 raft_consensus.cc:3037] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:02.272491 6206 raft_consensus.cc:3037] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:02.274219 6071 raft_consensus.cc:2454] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c725c7ecf4944cab607f1eb06de1772 in term 1.
I20260927 23:04:02.274554 5861 leader_election.cc:302] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c725c7ecf4944cab607f1eb06de1772, fd5ac701368640d7a33a2a9c550872f8; no voters:
I20260927 23:04:02.275327 6206 raft_consensus.cc:2454] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c725c7ecf4944cab607f1eb06de1772 in term 1.
I20260927 23:04:02.275269 6372 raft_consensus.cc:2781] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:02.275949 6372 raft_consensus.cc:686] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 LEADER]: Becoming Leader. State: Replica: 0c725c7ecf4944cab607f1eb06de1772, State: Running, Role: LEADER
I20260927 23:04:02.276255 6372 consensus_queue.cc:237] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } }
I20260927 23:04:02.278981 5771 catalog_manager.cc:5495] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "0c725c7ecf4944cab607f1eb06de1772" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c725c7ecf4944cab607f1eb06de1772" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 } health_report { overall_health: UNKNOWN } } }
I20260927 23:04:02.303244 6206 raft_consensus.cc:1261] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Refusing update from remote peer 0c725c7ecf4944cab607f1eb06de1772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:02.303287 6071 raft_consensus.cc:1261] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Refusing update from remote peer 0c725c7ecf4944cab607f1eb06de1772: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:02.303790 6386 consensus_queue.cc:1035] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd5ac701368640d7a33a2a9c550872f8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:02.304123 6372 consensus_queue.cc:1035] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fddd2fd7ccb14ddebc16f3b2c1a3dfaf" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 23:04:02.325268 6338 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260927 23:04:03.909233 5772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:03.909519 5772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:03.913546 5772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a05984665a814393b262309e52a01252: Sending DeleteTablet for 3 replicas of tablet 0d134c5a8c8a4d30a237a5c18b215310
I20260927 23:04:03.914213 5908 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d134c5a8c8a4d30a237a5c18b215310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:03 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57666
I20260927 23:04:03.914310 6047 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d134c5a8c8a4d30a237a5c18b215310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:03 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:59230
I20260927 23:04:03.914660 6186 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d134c5a8c8a4d30a237a5c18b215310 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:03 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41704
I20260927 23:04:03.914790 6418 tablet_replica.cc:330] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: stopping tablet replica
I20260927 23:04:03.915027 6418 raft_consensus.cc:2229] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:03.915026 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5843
I20260927 23:04:03.915508 6418 raft_consensus.cc:2258] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:03.916771 6418 ts_tablet_manager.cc:1905] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:03.923799 6418 ts_tablet_manager.cc:1918] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 23:04:03.923985 6418 log.cc:1196] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/0d134c5a8c8a4d30a237a5c18b215310
I20260927 23:04:03.924445 6418 ts_tablet_manager.cc:1939] T 0d134c5a8c8a4d30a237a5c18b215310 P 0c725c7ecf4944cab607f1eb06de1772: Deleting consensus metadata
I20260927 23:04:03.925501 5758 catalog_manager.cc:4841] TS 0c725c7ecf4944cab607f1eb06de1772 (127.3.240.193:37123): tablet 0d134c5a8c8a4d30a237a5c18b215310 (table test-table [id=69976631918242ebb6de4309dc43b632]) successfully deleted
I20260927 23:04:03.931810 6420 tablet_replica.cc:330] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: stopping tablet replica
I20260927 23:04:03.931985 6420 raft_consensus.cc:2229] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:03.932188 6420 raft_consensus.cc:2258] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:03.933135 6420 ts_tablet_manager.cc:1905] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:03.939646 6419 tablet_replica.cc:330] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: stopping tablet replica
I20260927 23:04:03.939877 6419 raft_consensus.cc:2229] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:03.940188 6419 raft_consensus.cc:2258] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:03.940204 6420 ts_tablet_manager.cc:1918] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 23:04:03.940382 6420 log.cc:1196] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/0d134c5a8c8a4d30a237a5c18b215310
I20260927 23:04:03.940905 6420 ts_tablet_manager.cc:1939] T 0d134c5a8c8a4d30a237a5c18b215310 P fddd2fd7ccb14ddebc16f3b2c1a3dfaf: Deleting consensus metadata
I20260927 23:04:03.942034 5756 catalog_manager.cc:4841] TS fddd2fd7ccb14ddebc16f3b2c1a3dfaf (127.3.240.195:45645): tablet 0d134c5a8c8a4d30a237a5c18b215310 (table test-table [id=69976631918242ebb6de4309dc43b632]) successfully deleted
I20260927 23:04:03.942210 6419 ts_tablet_manager.cc:1905] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:03.947278 6419 ts_tablet_manager.cc:1918] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260927 23:04:03.947408 6419 log.cc:1196] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/0d134c5a8c8a4d30a237a5c18b215310
I20260927 23:04:03.947852 6419 ts_tablet_manager.cc:1939] T 0d134c5a8c8a4d30a237a5c18b215310 P fd5ac701368640d7a33a2a9c550872f8: Deleting consensus metadata
Sep 27 23:04:03 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:04:03.948719 5758 catalog_manager.cc:4841] TS fd5ac701368640d7a33a2a9c550872f8 (127.3.240.194:46785): tablet 0d134c5a8c8a4d30a237a5c18b215310 (table test-table [id=69976631918242ebb6de4309dc43b632]) successfully deleted
I20260927 23:04:03.953436 6422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:03.918177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57694 (local address 127.3.240.193:37123)
0927 23:04:03.939227 (+ 21050us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:03.939234 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:03.939259 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:04:03.939458 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:03.939463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:03.939552 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:03.939645 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:03.941668 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:03.942619 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:03.943974 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:03.944407 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:03.944554 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:03.948777 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:03.948794 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:03.948816 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:03.948855 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:03.951409 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:03.951959 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:03.951965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:03.951968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:03.952092 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:03.952380 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:03.952383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:03.952385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:03.952670 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:03.952859 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:03.953050 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 23:04:03.953200 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20928,"thread_start_us":21285,"threads_started":1}
W20260927 23:04:03.953996 5973 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57694
W20260927 23:04:03.954602 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5843: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:04:03.954865 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 5843
I20260927 23:04:03.981686 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5982
Sep 27 23:04:03 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:04:03.998096 6423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:03.982391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.3.240.194:46785)
0927 23:04:03.982906 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:03.982910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:03.982927 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:04:03.982972 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:03.982975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:03.983094 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:03.983181 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:03.984817 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:03.985681 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:03.987452 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:03.987979 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:03.988132 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:03.992687 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:03.992699 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:03.992715 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:03.992747 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:03.995642 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:03.996313 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:03.996321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:03.996323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:03.996462 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:03.996819 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:03.996829 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:03.996833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:03.997152 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:03.997422 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:03.997694 (+ 272us) server_negotiation.cc:299] Negotiation successful
0927 23:04:03.997841 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":251,"threads_started":1}
W20260927 23:04:03.998708 6112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59264
W20260927 23:04:03.999310 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5982: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:04:03.999425 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 5982
I20260927 23:04:04.022719 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6121
Sep 27 23:04:04 dist-test-slave-lmfp krb5kdc[5727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550241, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:04:04.037708 6424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:04.023412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.3.240.195:45645)
0927 23:04:04.023855 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:04.023859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:04.023886 (+ 27us) server_negotiation.cc:407] Connection header received
0927 23:04:04.023934 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:04.023937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:04.023995 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:04.024096 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:04.025462 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:04.026362 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:04.027622 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:04.028019 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:04.028142 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:04.032139 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:04.032154 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:04.032171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:04.032204 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:04.035256 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:04.035827 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:04.035835 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:04.035837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:04.035961 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:04.036264 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:04.036267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:04.036269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:04.036671 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:04.036886 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:04.037138 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 23:04:04.037300 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":211,"threads_started":1}
W20260927 23:04:04.038457 6251 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41728
W20260927 23:04:04.039108 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6121: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:04:04.039296 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 6121
I20260927 23:04:04.060191 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 5740
I20260927 23:04:04.061322 5812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:04.172505 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 5740
2026-09-27T23:04:04Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7915 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:04 dist-test-slave-lmfp krb5kdc[6428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:04 dist-test-slave-lmfp krb5kdc[6428](info): set up 2 sockets
Sep 27 23:04:04 dist-test-slave-lmfp krb5kdc[6428](info): commencing operation
krb5kdc: starting...
W20260927 23:04:05.207391 4035 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:05 dist-test-slave-lmfp krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:04:05.383054 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37835
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:05.689133 6441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:05.689445 6441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:05.689601 6441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:05.696280 6441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:05.696411 6441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:05.696507 6441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:05.696589 6441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:05.696662 6441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:05.705821 6441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:05.707078 6441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:05.708304 6441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:05.719413 6446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:05.719467 6449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:05.719507 6447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:05.720230 6441 server_base.cc:1034] running on GCE node
I20260927 23:04:05.720932 6441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:05.722208 6441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:05.723431 6441 hybrid_clock.cc:648] HybridClock initialized: now 1790550245723388 us; error 50 us; skew 500 ppm
I20260927 23:04:05.723773 6441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:05 dist-test-slave-lmfp krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550245, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:05.726555 6441 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:04:05.727883 6441 webserver.cc:460] Webserver started at http://127.3.240.254:46167/ using document root <none> and password file <none>
I20260927 23:04:05.728523 6441 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:05.728654 6441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:05.728931 6441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:05.731653 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "b8c149b011e9461f84c5bff8f901445f"
format_stamp: "Formatted at 2026-09-27 23:04:05 on dist-test-slave-lmfp"
server_key: "0fe3b638a998e63ff258a284b5622ad3"
server_key_iv: "81e9d2766ca526e1111a1e0f1344e653"
server_key_version: "encryptionkey@0"
I20260927 23:04:05.732420 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "b8c149b011e9461f84c5bff8f901445f"
format_stamp: "Formatted at 2026-09-27 23:04:05 on dist-test-slave-lmfp"
server_key: "0fe3b638a998e63ff258a284b5622ad3"
server_key_iv: "81e9d2766ca526e1111a1e0f1344e653"
server_key_version: "encryptionkey@0"
I20260927 23:04:05.737020 6441 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:04:05.740367 6456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:05.741494 6441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:04:05.741658 6441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "b8c149b011e9461f84c5bff8f901445f"
format_stamp: "Formatted at 2026-09-27 23:04:05 on dist-test-slave-lmfp"
server_key: "0fe3b638a998e63ff258a284b5622ad3"
server_key_iv: "81e9d2766ca526e1111a1e0f1344e653"
server_key_version: "encryptionkey@0"
I20260927 23:04:05.741829 6441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:05.758069 6441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:05.761595 6441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:05.761859 6441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:05.779240 6441 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:37835
I20260927 23:04:05.779278 6507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:37835 every 8 connection(s)
I20260927 23:04:05.780511 6441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:05.783954 6508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:05.788754 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 6441
I20260927 23:04:05.788949 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:05.789502 4035 external_mini_cluster.cc:1428] Setting key 25c99c1283b2cc15d87288ae9f4800f9
I20260927 23:04:05.794476 6508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f: Bootstrap starting.
I20260927 23:04:05.797848 6508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:05.798902 6508 log.cc:824] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:05.801614 6508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f: No bootstrap required, opened a new log
Sep 27 23:04:05 dist-test-slave-lmfp krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:05.810328 6508 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c149b011e9461f84c5bff8f901445f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37835 } }
I20260927 23:04:05.810761 6508 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:05.810930 6508 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c149b011e9461f84c5bff8f901445f, State: Initialized, Role: FOLLOWER
I20260927 23:04:05.811550 6508 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c149b011e9461f84c5bff8f901445f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37835 } }
I20260927 23:04:05.811802 6508 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:05.811941 6508 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:05.812120 6508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:05.812260 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:05.791377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.3.240.254:37835)
0927 23:04:05.792216 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:05.792231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:05.792278 (+ 47us) server_negotiation.cc:407] Connection header received
0927 23:04:05.794441 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:05.794482 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:05.794894 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:05.795291 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:05.796572 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:05.798494 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:05.799859 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:05.800764 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:05.800928 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:05.805065 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:05.805075 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:05.805099 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:05.805153 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:05.808154 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:05.808924 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:05.808929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:05.808935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:05.809076 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:05.809445 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:05.809448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:05.809449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:05.809911 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:05.810100 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:05.810547 (+ 447us) server_negotiation.cc:299] Negotiation successful
0927 23:04:05.810986 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":344,"threads_started":1}
I20260927 23:04:05.814431 6508 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c149b011e9461f84c5bff8f901445f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37835 } }
I20260927 23:04:05.814950 6508 leader_election.cc:302] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8c149b011e9461f84c5bff8f901445f; no voters:
I20260927 23:04:05.815528 6508 leader_election.cc:288] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:05.815661 6513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:05.816330 6513 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [term 1 LEADER]: Becoming Leader. State: Replica: b8c149b011e9461f84c5bff8f901445f, State: Running, Role: LEADER
I20260927 23:04:05.816807 6513 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c149b011e9461f84c5bff8f901445f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37835 } }
I20260927 23:04:05.817371 6508 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:05.819465 6514 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8c149b011e9461f84c5bff8f901445f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c149b011e9461f84c5bff8f901445f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37835 } } }
I20260927 23:04:05.819818 6514 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:05.819582 6515 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8c149b011e9461f84c5bff8f901445f. Latest consensus state: current_term: 1 leader_uuid: "b8c149b011e9461f84c5bff8f901445f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8c149b011e9461f84c5bff8f901445f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37835 } } }
I20260927 23:04:05.820144 6515 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:05.822938 6518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:05.827751 6518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:05.836239 6518 catalog_manager.cc:1347] Generated new cluster ID: 9d9a734c882b48cd88509e07510fb281
I20260927 23:04:05.836339 6518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:05.863332 6518 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:05.865124 6518 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:05.874204 6518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8c149b011e9461f84c5bff8f901445f: Generated new TSK 0
I20260927 23:04:05.875180 6518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:04:05.925508 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37835
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:06.231412 6534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:06.231671 6534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:06.231761 6534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:06.238543 6534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:06.238632 6534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:06.238734 6534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:06.247831 6534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:06.248943 6534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:06.250089 6534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:06.262630 6539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:06.262636 6542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:06.262878 6540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:06.263623 6534 server_base.cc:1034] running on GCE node
I20260927 23:04:06.264192 6534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:06.265159 6534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:06.266350 6534 hybrid_clock.cc:648] HybridClock initialized: now 1790550246266316 us; error 28 us; skew 500 ppm
I20260927 23:04:06.266659 6534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:06 dist-test-slave-lmfp krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:06.269766 6534 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:04:06.271129 6534 webserver.cc:460] Webserver started at http://127.3.240.193:35969/ using document root <none> and password file <none>
I20260927 23:04:06.271736 6534 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:06.271824 6534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:06.272058 6534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:06.274861 6534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "3ccc41e2159042bfa240efe528a2dbce"
format_stamp: "Formatted at 2026-09-27 23:04:06 on dist-test-slave-lmfp"
server_key: "267ae80b31e3d567572c3b03f30a4210"
server_key_iv: "3bd94de92fc4944884d45ff52ecc785b"
server_key_version: "encryptionkey@0"
I20260927 23:04:06.275642 6534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "3ccc41e2159042bfa240efe528a2dbce"
format_stamp: "Formatted at 2026-09-27 23:04:06 on dist-test-slave-lmfp"
server_key: "267ae80b31e3d567572c3b03f30a4210"
server_key_iv: "3bd94de92fc4944884d45ff52ecc785b"
server_key_version: "encryptionkey@0"
I20260927 23:04:06.280356 6534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 23:04:06.283463 6549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:06.284725 6534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 23:04:06.284896 6534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "3ccc41e2159042bfa240efe528a2dbce"
format_stamp: "Formatted at 2026-09-27 23:04:06 on dist-test-slave-lmfp"
server_key: "267ae80b31e3d567572c3b03f30a4210"
server_key_iv: "3bd94de92fc4944884d45ff52ecc785b"
server_key_version: "encryptionkey@0"
I20260927 23:04:06.285092 6534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:06.301012 6534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:06.304971 6534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:06.305241 6534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:06.306121 6534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:06.307360 6534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:06.307459 6534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:06.307593 6534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:06.307663 6534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:06.340598 6534 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:35571
I20260927 23:04:06.340624 6661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:35571 every 8 connection(s)
I20260927 23:04:06.341856 6534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:06.351042 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 6534
I20260927 23:04:06.351280 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:06.351887 4035 external_mini_cluster.cc:1428] Setting key 0c50c2211bc9ff4d7d061129d920683a
Sep 27 23:04:06 dist-test-slave-lmfp krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:06.361214 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:06.344225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:53785 (local address 127.3.240.254:37835)
0927 23:04:06.344642 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:06.344647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:06.345629 (+ 982us) server_negotiation.cc:407] Connection header received
0927 23:04:06.347537 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:06.347542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:06.347628 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:06.347745 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:06.349166 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:06.349905 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:06.351591 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:06.352052 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:06.352189 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:06.355808 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:06.355813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:06.355816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:06.355844 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:06.358352 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:06.359058 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:06.359062 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:06.359065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:06.359168 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:06.359592 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:06.359597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:06.359600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:06.359850 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:06.360007 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:06.360635 (+ 628us) server_negotiation.cc:299] Negotiation successful
0927 23:04:06.360831 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":241,"threads_started":1}
I20260927 23:04:06.362136 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:06.344693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37835 (local address 127.3.240.193:53785)
0927 23:04:06.345470 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:06.345508 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:06.347268 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:06.347983 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:06.347991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:06.348476 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:06.349006 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:06.349022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:06.350071 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:06.350074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:06.351452 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:06.351464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:06.352275 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:06.352285 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:06.352538 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:06.353176 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:06.353209 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:06.355599 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:06.358493 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:06.358501 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:06.358516 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:06.358902 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:06.359270 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:06.359276 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:06.359280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:06.359465 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:06.359972 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:06.359980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:06.360300 (+ 320us) client_negotiation.cc:769] Sending connection context
0927 23:04:06.360555 (+ 255us) client_negotiation.cc:240] Negotiation successful
0927 23:04:06.361058 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":329,"threads_started":1}
I20260927 23:04:06.363657 6662 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37835
I20260927 23:04:06.363976 6662 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:06.364835 6662 heartbeater.cc:511] Master 127.3.240.254:37835 requested a full tablet report, sending...
I20260927 23:04:06.367430 6473 ts_manager.cc:194] Registered new tserver with Master: 3ccc41e2159042bfa240efe528a2dbce (127.3.240.193:35571)
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260927 23:04:06.369498 6473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:53785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:04:06.394124 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37835
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:06.704409 6668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:06.704674 6668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:06.704768 6668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:06.711578 6668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:06.711681 6668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:06.711791 6668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:06.721293 6668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:06.722463 6668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:06.723810 6668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:06.736563 6673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:06.736469 6676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:06.736471 6674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:06.737144 6668 server_base.cc:1034] running on GCE node
I20260927 23:04:06.737828 6668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:06.738768 6668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:06.739957 6668 hybrid_clock.cc:648] HybridClock initialized: now 1790550246739898 us; error 53 us; skew 500 ppm
I20260927 23:04:06.740270 6668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:06 dist-test-slave-lmfp krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:06.743407 6668 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:04:06.744930 6668 webserver.cc:460] Webserver started at http://127.3.240.194:36735/ using document root <none> and password file <none>
I20260927 23:04:06.745801 6668 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:06.745942 6668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:06.746281 6668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:06.749022 6668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "aaa98ba9a5b045948250963c01172b5b"
format_stamp: "Formatted at 2026-09-27 23:04:06 on dist-test-slave-lmfp"
server_key: "813626822bb548f83e1625ab6bd63111"
server_key_iv: "0650954e3f1e2788e184b317b6d70b43"
server_key_version: "encryptionkey@0"
I20260927 23:04:06.749825 6668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "aaa98ba9a5b045948250963c01172b5b"
format_stamp: "Formatted at 2026-09-27 23:04:06 on dist-test-slave-lmfp"
server_key: "813626822bb548f83e1625ab6bd63111"
server_key_iv: "0650954e3f1e2788e184b317b6d70b43"
server_key_version: "encryptionkey@0"
I20260927 23:04:06.754657 6668 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 23:04:06.758090 6683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:06.759557 6668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:06.759720 6668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "aaa98ba9a5b045948250963c01172b5b"
format_stamp: "Formatted at 2026-09-27 23:04:06 on dist-test-slave-lmfp"
server_key: "813626822bb548f83e1625ab6bd63111"
server_key_iv: "0650954e3f1e2788e184b317b6d70b43"
server_key_version: "encryptionkey@0"
I20260927 23:04:06.759956 6668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:06.786881 6668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:06.791463 6668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:06.791733 6668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:06.792622 6668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:06.794178 6668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:06.794263 6668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:06.794438 6668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:06.794510 6668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:06.826321 6668 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:39971
I20260927 23:04:06.826404 6795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:39971 every 8 connection(s)
I20260927 23:04:06.827622 6668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:06.831295 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 6668
I20260927 23:04:06.831498 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:06.832026 4035 external_mini_cluster.cc:1428] Setting key ab1c0ca8019f62d2143c0f8141fc1b3b
Sep 27 23:04:06 dist-test-slave-lmfp krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550246, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:06.847965 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:06.830083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:39171 (local address 127.3.240.254:37835)
0927 23:04:06.830250 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:06.830254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:06.831603 (+ 1349us) server_negotiation.cc:407] Connection header received
0927 23:04:06.833857 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:06.833861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:06.833949 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:06.834052 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:06.835543 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:06.836224 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:06.837738 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:06.838383 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:06.838523 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:06.842269 (+ 3746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:06.842274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:06.842278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:06.842307 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:06.844850 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:06.845673 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:06.845679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:06.845684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:06.845809 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:06.846222 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:06.846227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:06.846231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:06.846553 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:06.846708 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:06.847416 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 23:04:06.847628 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260927 23:04:06.848948 6798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:06.830548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37835 (local address 127.3.240.194:39171)
0927 23:04:06.831400 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:06.831482 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:06.833569 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:06.834278 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:06.834285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:06.834772 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:06.835378 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:06.835395 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:06.836366 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:06.836376 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:06.837589 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:06.837600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:06.838584 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:06.838589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:06.838782 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:06.839465 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:06.839485 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:06.842050 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:06.844977 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:06.844987 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:06.845002 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:06.845545 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:06.845928 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:06.845932 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:06.845936 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:06.846109 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:06.846685 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:06.846693 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:06.847068 (+ 375us) client_negotiation.cc:769] Sending connection context
0927 23:04:06.847319 (+ 251us) client_negotiation.cc:240] Negotiation successful
0927 23:04:06.847826 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":294,"threads_started":1}
I20260927 23:04:06.850371 6796 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37835
I20260927 23:04:06.850690 6796 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
I20260927 23:04:06.851451 6796 heartbeater.cc:511] Master 127.3.240.254:37835 requested a full tablet report, sending...
I20260927 23:04:06.852929 6473 ts_manager.cc:194] Registered new tserver with Master: aaa98ba9a5b045948250963c01172b5b (127.3.240.194:39971)
I20260927 23:04:06.854130 6473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:39171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:04:06.873821 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37835
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:07.179364 6801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:07.179612 6801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:07.179700 6801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:07.186523 6801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:07.186611 6801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:07.186708 6801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:07.196077 6801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37835
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:07.197182 6801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:07.198468 6801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:07.211071 6806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:07.211015 6807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:07.211007 6809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:07.211844 6801 server_base.cc:1034] running on GCE node
I20260927 23:04:07.212383 6801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:07.213251 6801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:07.214493 6801 hybrid_clock.cc:648] HybridClock initialized: now 1790550247214459 us; error 40 us; skew 500 ppm
I20260927 23:04:07.214790 6801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:07 dist-test-slave-lmfp krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:07.226514 6801 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:04:07.228022 6801 webserver.cc:460] Webserver started at http://127.3.240.195:41933/ using document root <none> and password file <none>
I20260927 23:04:07.228724 6801 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:07.228857 6801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:07.229149 6801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:07.231871 6801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "aba6c9a7ed0745b99af18c06d84c6f05"
format_stamp: "Formatted at 2026-09-27 23:04:07 on dist-test-slave-lmfp"
server_key: "5663544a70bda74e5674571f9a2c8edb"
server_key_iv: "cc868492f4d3b8c76a07ca97b4caa58a"
server_key_version: "encryptionkey@0"
I20260927 23:04:07.232573 6801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "aba6c9a7ed0745b99af18c06d84c6f05"
format_stamp: "Formatted at 2026-09-27 23:04:07 on dist-test-slave-lmfp"
server_key: "5663544a70bda74e5674571f9a2c8edb"
server_key_iv: "cc868492f4d3b8c76a07ca97b4caa58a"
server_key_version: "encryptionkey@0"
I20260927 23:04:07.237151 6801 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:04:07.240324 6816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:07.241485 6801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:04:07.241636 6801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "aba6c9a7ed0745b99af18c06d84c6f05"
format_stamp: "Formatted at 2026-09-27 23:04:07 on dist-test-slave-lmfp"
server_key: "5663544a70bda74e5674571f9a2c8edb"
server_key_iv: "cc868492f4d3b8c76a07ca97b4caa58a"
server_key_version: "encryptionkey@0"
I20260927 23:04:07.241801 6801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:07.257963 6801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:07.262126 6801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:07.262394 6801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:07.263446 6801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:07.264693 6801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:07.264791 6801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:07.264938 6801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:07.265012 6801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:07.296005 6801 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:40389
I20260927 23:04:07.296041 6928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:40389 every 8 connection(s)
I20260927 23:04:07.297155 6801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:07.299247 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 6801
I20260927 23:04:07.299440 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:07.299960 4035 external_mini_cluster.cc:1428] Setting key 7c497e605a978d647c5e7d35b006a4f1
Sep 27 23:04:07 dist-test-slave-lmfp krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550247, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:07.316437 6664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:07.299644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:42279 (local address 127.3.240.254:37835)
0927 23:04:07.299855 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:07.299860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:07.301255 (+ 1395us) server_negotiation.cc:407] Connection header received
0927 23:04:07.303256 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:07.303261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:07.303341 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:07.303451 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:07.304844 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:07.305569 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:07.306936 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:07.307383 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:07.307514 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:07.310649 (+ 3135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:07.310654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:07.310657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:07.310686 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:07.313158 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:07.313856 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:07.313860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:07.313862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:07.314027 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:07.314566 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:07.314571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:07.314576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:07.314978 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:07.315178 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:07.315874 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 23:04:07.316107 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260927 23:04:07.317477 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:07.300292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37835 (local address 127.3.240.195:42279)
0927 23:04:07.301116 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:07.301166 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:07.302963 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:07.303698 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:07.303706 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:07.304167 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:07.304689 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:07.304705 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:07.305715 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:07.305726 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:07.306803 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:07.306813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:07.307639 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:07.307646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:07.307855 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:07.308412 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:07.308436 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:07.310426 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:07.313291 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:07.313299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:07.313315 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:07.313750 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:07.314171 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:07.314175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:07.314178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:07.314406 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:07.315130 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:07.315139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:07.315546 (+ 407us) client_negotiation.cc:769] Sending connection context
0927 23:04:07.315807 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 23:04:07.316290 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":319,"threads_started":1}
I20260927 23:04:07.318717 6929 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37835
I20260927 23:04:07.319048 6929 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:07.319737 6929 heartbeater.cc:511] Master 127.3.240.254:37835 requested a full tablet report, sending...
I20260927 23:04:07.321048 6473 ts_manager.cc:194] Registered new tserver with Master: aba6c9a7ed0745b99af18c06d84c6f05 (127.3.240.195:40389)
I20260927 23:04:07.321815 6473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:42279
I20260927 23:04:07.327471 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 23:04:07.356561 6664 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:07.354202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45650 (local address 127.3.240.254:37835)
0927 23:04:07.354358 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:07.354362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:07.354937 (+ 575us) server_negotiation.cc:407] Connection header received
0927 23:04:07.355163 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:07.355167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:07.355221 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:07.355308 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:07.356387 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45650: BlockingRecv error: recv got EOF from 127.0.0.1:45650 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 23:04:07.360195 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6534
W20260927 23:04:07.362536 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6534: Not authorized: Client connection negotiation failed: client connection to 127.3.240.193:35571: server requires authentication, but client does not have Kerberos credentials available
I20260927 23:04:07.362728 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 6534
I20260927 23:04:07.377780 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6668
W20260927 23:04:07.380317 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6668: Not authorized: Client connection negotiation failed: client connection to 127.3.240.194:39971: server requires authentication, but client does not have Kerberos credentials available
I20260927 23:04:07.380487 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 6668
I20260927 23:04:07.394865 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6801
W20260927 23:04:07.397586 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6801: Not authorized: Client connection negotiation failed: client connection to 127.3.240.195:40389: server requires authentication, but client does not have Kerberos credentials available
I20260927 23:04:07.397692 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 6801
I20260927 23:04:07.411896 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6441
I20260927 23:04:07.412921 6503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:07.508533 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 6441
2026-09-27T23:04:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3334 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:07 dist-test-slave-lmfp krb5kdc[6946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:07 dist-test-slave-lmfp krb5kdc[6946](info): set up 2 sockets
Sep 27 23:04:07 dist-test-slave-lmfp krb5kdc[6946](info): commencing operation
krb5kdc: starting...
W20260927 23:04:08.550896 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:08 dist-test-slave-lmfp krb5kdc[6946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:04:08.720252 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37119
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:09.012863 6959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:09.013199 6959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:09.013350 6959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:09.020287 6959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:09.020416 6959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:09.020511 6959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:09.020617 6959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:09.020700 6959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:09.030221 6959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:09.031491 6959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:09.032718 6959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:09.044104 6964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:09.044152 6965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:09.044157 6967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:09.044948 6959 server_base.cc:1034] running on GCE node
I20260927 23:04:09.045616 6959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:09.046852 6959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:09.048056 6959 hybrid_clock.cc:648] HybridClock initialized: now 1790550249048028 us; error 62 us; skew 500 ppm
I20260927 23:04:09.048403 6959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:09 dist-test-slave-lmfp krb5kdc[6946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550249, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:09.051373 6959 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:04:09.052734 6959 webserver.cc:460] Webserver started at http://127.3.240.254:39493/ using document root <none> and password file <none>
I20260927 23:04:09.053380 6959 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:09.053498 6959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:09.053769 6959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:09.056458 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "599d0ad619ec443e94ec08a64067a2be"
format_stamp: "Formatted at 2026-09-27 23:04:09 on dist-test-slave-lmfp"
server_key: "4edb9371dcecdb4bff9f2f636de5aff2"
server_key_iv: "342da460dcafe21d20353920ae2f45b0"
server_key_version: "encryptionkey@0"
I20260927 23:04:09.057135 6959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "599d0ad619ec443e94ec08a64067a2be"
format_stamp: "Formatted at 2026-09-27 23:04:09 on dist-test-slave-lmfp"
server_key: "4edb9371dcecdb4bff9f2f636de5aff2"
server_key_iv: "342da460dcafe21d20353920ae2f45b0"
server_key_version: "encryptionkey@0"
I20260927 23:04:09.061756 6959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 23:04:09.065009 6974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:09.066159 6959 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:04:09.066318 6959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "599d0ad619ec443e94ec08a64067a2be"
format_stamp: "Formatted at 2026-09-27 23:04:09 on dist-test-slave-lmfp"
server_key: "4edb9371dcecdb4bff9f2f636de5aff2"
server_key_iv: "342da460dcafe21d20353920ae2f45b0"
server_key_version: "encryptionkey@0"
I20260927 23:04:09.066479 6959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:09.094306 6959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:09.098392 6959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:09.098662 6959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:09.119441 6959 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:37119
I20260927 23:04:09.119469 7025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:37119 every 8 connection(s)
I20260927 23:04:09.120744 6959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:09.124315 7026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:09.125119 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 6959
I20260927 23:04:09.125337 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:09.126266 4035 external_mini_cluster.cc:1428] Setting key 64f1b95bf6c6f161d5b5054947cf85d8
I20260927 23:04:09.138095 7026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be: Bootstrap starting.
Sep 27 23:04:09 dist-test-slave-lmfp krb5kdc[6946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:09.141522 7026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:09.142508 7026 log.cc:824] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:09.145191 7026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be: No bootstrap required, opened a new log
I20260927 23:04:09.148245 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:09.128214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47294 (local address 127.3.240.254:37119)
0927 23:04:09.128950 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:09.128960 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:09.129010 (+ 50us) server_negotiation.cc:407] Connection header received
0927 23:04:09.130664 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:09.130685 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:09.131128 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:09.131454 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:09.133127 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:09.134316 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:09.136017 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:09.136636 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:09.136758 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:09.141178 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:09.141185 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:09.141198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:09.141231 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:09.144469 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:09.145106 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:09.145120 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:09.145125 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:09.145308 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:09.145733 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:09.145736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:09.145738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:09.146213 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:09.146424 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:09.146846 (+ 422us) server_negotiation.cc:299] Negotiation successful
0927 23:04:09.147336 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":364,"threads_started":1}
I20260927 23:04:09.151707 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:04:09.153302 7026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599d0ad619ec443e94ec08a64067a2be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37119 } }
I20260927 23:04:09.153633 7026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:09.153735 7026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 599d0ad619ec443e94ec08a64067a2be, State: Initialized, Role: FOLLOWER
I20260927 23:04:09.154295 7026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599d0ad619ec443e94ec08a64067a2be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37119 } }
I20260927 23:04:09.154521 7026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:09.154670 7026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:09.154842 7026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:09.156788 7026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599d0ad619ec443e94ec08a64067a2be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37119 } }
I20260927 23:04:09.157238 7026 leader_election.cc:302] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 599d0ad619ec443e94ec08a64067a2be; no voters:
I20260927 23:04:09.157894 7026 leader_election.cc:288] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:09.158016 7032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:09.159746 7026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:09.164021 7032 raft_consensus.cc:686] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [term 1 LEADER]: Becoming Leader. State: Replica: 599d0ad619ec443e94ec08a64067a2be, State: Running, Role: LEADER
I20260927 23:04:09.164656 7032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599d0ad619ec443e94ec08a64067a2be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37119 } }
I20260927 23:04:09.168458 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "599d0ad619ec443e94ec08a64067a2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599d0ad619ec443e94ec08a64067a2be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37119 } } }
I20260927 23:04:09.168530 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 599d0ad619ec443e94ec08a64067a2be. Latest consensus state: current_term: 1 leader_uuid: "599d0ad619ec443e94ec08a64067a2be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "599d0ad619ec443e94ec08a64067a2be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37119 } } }
I20260927 23:04:09.168836 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:09.168845 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be [sys.catalog]: This master's current role is: LEADER
W20260927 23:04:09.175539 7047 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:04:09.175669 7047 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:04:09.175864 7046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:09.177831 7046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:09.186879 7046 catalog_manager.cc:1347] Generated new cluster ID: f99b026ec9a047ae88f0434f4a168712
I20260927 23:04:09.187019 7046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:09.203311 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:09.186743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47304 (local address 127.3.240.254:37119)
0927 23:04:09.186886 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:09.186891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:09.195606 (+ 8715us) server_negotiation.cc:407] Connection header received
0927 23:04:09.195783 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:09.195788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:09.195874 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:09.195977 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:09.197164 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:09.198229 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:09.199913 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:09.200531 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:09.200646 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:09.201784 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:09.201790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:09.201794 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:09.202509 (+ 715us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:09.202583 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:09.202612 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:09.202737 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:09.202887 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 23:04:09.202970 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":31}
I20260927 23:04:09.210614 7046 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:09.212082 7046 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:09.222404 7046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 599d0ad619ec443e94ec08a64067a2be: Generated new TSK 0
I20260927 23:04:09.223285 7046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:04:09.234779 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 6959
I20260927 23:04:09.236341 7021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:09.322459 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 6959
2026-09-27T23:04:09Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1808 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:09 dist-test-slave-lmfp krb5kdc[7060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:09 dist-test-slave-lmfp krb5kdc[7060](info): set up 2 sockets
Sep 27 23:04:09 dist-test-slave-lmfp krb5kdc[7060](info): commencing operation
krb5kdc: starting...
W20260927 23:04:10.359367 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:10 dist-test-slave-lmfp krb5kdc[7060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:04:10.534255 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35917
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:35917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:10.843315 7073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:10.843567 7073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:10.843755 7073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:10.850181 7073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:10.850283 7073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:10.850384 7073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:10.850467 7073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:10.850546 7073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:10.859457 7073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:35917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:10.860635 7073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:10.861836 7073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:10.872736 7079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:10.872808 7078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:10.872833 7081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:10.873728 7073 server_base.cc:1034] running on GCE node
I20260927 23:04:10.874325 7073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:10.875576 7073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:10.876778 7073 hybrid_clock.cc:648] HybridClock initialized: now 1790550250876710 us; error 46 us; skew 500 ppm
I20260927 23:04:10.877085 7073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:10 dist-test-slave-lmfp krb5kdc[7060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550250, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:10.879905 7073 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:04:10.881206 7073 webserver.cc:460] Webserver started at http://127.3.240.254:41215/ using document root <none> and password file <none>
I20260927 23:04:10.881796 7073 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:10.881933 7073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:10.882220 7073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:10.885056 7073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "8a348b41e6974b618b1e9eaff1789bbc"
format_stamp: "Formatted at 2026-09-27 23:04:10 on dist-test-slave-lmfp"
server_key: "15f9812b3b7e6705b10acd2dc2157e9f"
server_key_iv: "1ec1093ee43403b8e88c9ed67c73cee4"
server_key_version: "encryptionkey@0"
I20260927 23:04:10.885746 7073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "8a348b41e6974b618b1e9eaff1789bbc"
format_stamp: "Formatted at 2026-09-27 23:04:10 on dist-test-slave-lmfp"
server_key: "15f9812b3b7e6705b10acd2dc2157e9f"
server_key_iv: "1ec1093ee43403b8e88c9ed67c73cee4"
server_key_version: "encryptionkey@0"
I20260927 23:04:10.890676 7073 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260927 23:04:10.894033 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:10.895416 7073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:10.895607 7073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "8a348b41e6974b618b1e9eaff1789bbc"
format_stamp: "Formatted at 2026-09-27 23:04:10 on dist-test-slave-lmfp"
server_key: "15f9812b3b7e6705b10acd2dc2157e9f"
server_key_iv: "1ec1093ee43403b8e88c9ed67c73cee4"
server_key_version: "encryptionkey@0"
I20260927 23:04:10.895826 7073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:10.913618 7073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:10.917533 7073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:10.917809 7073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:10.935214 7073 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:35917
I20260927 23:04:10.935245 7139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:35917 every 8 connection(s)
I20260927 23:04:10.936447 7073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:10.939610 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 7073
I20260927 23:04:10.939816 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:10.940164 7140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:10.940647 4035 external_mini_cluster.cc:1428] Setting key 3fd3ab0111544d2f9b20e707e83f54b5
I20260927 23:04:10.953907 7140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc: Bootstrap starting.
Sep 27 23:04:10 dist-test-slave-lmfp krb5kdc[7060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:10.957273 7140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:10.958534 7140 log.cc:824] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:10.961777 7140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc: No bootstrap required, opened a new log
I20260927 23:04:10.963333 7143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:10.942587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.3.240.254:35917)
0927 23:04:10.943433 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:10.943443 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:10.943483 (+ 40us) server_negotiation.cc:407] Connection header received
0927 23:04:10.945017 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:10.945051 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:10.945398 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:10.945713 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:10.947365 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:10.948520 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:10.950242 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:10.950867 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:10.950976 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:10.955365 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:10.955374 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:10.955398 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:10.955442 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:10.958942 (+ 3500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:10.959794 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:10.959800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:10.959805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:10.960019 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:10.960406 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:10.960410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:10.960413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:10.961081 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:10.961383 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:10.961620 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 23:04:10.962013 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":445,"threads_started":1}
I20260927 23:04:10.969364 7140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a348b41e6974b618b1e9eaff1789bbc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35917 } }
I20260927 23:04:10.969652 7140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:10.969722 7140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a348b41e6974b618b1e9eaff1789bbc, State: Initialized, Role: FOLLOWER
I20260927 23:04:10.970237 7140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a348b41e6974b618b1e9eaff1789bbc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35917 } }
I20260927 23:04:10.970409 7140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:10.970502 7140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:10.970613 7140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:10.972524 7140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a348b41e6974b618b1e9eaff1789bbc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35917 } }
I20260927 23:04:10.972910 7140 leader_election.cc:302] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a348b41e6974b618b1e9eaff1789bbc; no voters:
I20260927 23:04:10.973364 7140 leader_election.cc:288] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:10.973501 7145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:10.974212 7145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [term 1 LEADER]: Becoming Leader. State: Replica: 8a348b41e6974b618b1e9eaff1789bbc, State: Running, Role: LEADER
I20260927 23:04:10.974679 7145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a348b41e6974b618b1e9eaff1789bbc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35917 } }
I20260927 23:04:10.975361 7140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:10.977749 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a348b41e6974b618b1e9eaff1789bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a348b41e6974b618b1e9eaff1789bbc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35917 } } }
I20260927 23:04:10.977739 7147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a348b41e6974b618b1e9eaff1789bbc. Latest consensus state: current_term: 1 leader_uuid: "8a348b41e6974b618b1e9eaff1789bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a348b41e6974b618b1e9eaff1789bbc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35917 } } }
I20260927 23:04:10.978123 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:10.978142 7147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:10.984324 7152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:10.989043 7152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:10.996308 7152 catalog_manager.cc:1347] Generated new cluster ID: 8c0009af76f640dab05b354cb6415b89
I20260927 23:04:10.996402 7152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:11.006182 7152 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:11.007395 7152 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:11.018100 7152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a348b41e6974b618b1e9eaff1789bbc: Generated new TSK 0
I20260927 23:04:11.018914 7152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:04:11.068863 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35917
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:11.381373 7166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:11.381630 7166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:11.381788 7166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:11.389564 7166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:11.389683 7166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:11.389854 7166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:11.398979 7166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:11.400399 7166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:11.401669 7166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:11.414315 7172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:11.414335 7171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:11.414538 7174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:11.415077 7166 server_base.cc:1034] running on GCE node
I20260927 23:04:11.415697 7166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:11.416501 7166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:11.417766 7166 hybrid_clock.cc:648] HybridClock initialized: now 1790550251417730 us; error 53 us; skew 500 ppm
I20260927 23:04:11.418045 7166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:11 dist-test-slave-lmfp krb5kdc[7060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:11.421218 7166 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:04:11.422837 7166 webserver.cc:460] Webserver started at http://127.3.240.193:45815/ using document root <none> and password file <none>
I20260927 23:04:11.423580 7166 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:11.423681 7166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:11.424002 7166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:11.426792 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "1162e642335f4a26b9521c5f786ceeb3"
format_stamp: "Formatted at 2026-09-27 23:04:11 on dist-test-slave-lmfp"
server_key: "5cb34122e55294c3aa1c7da38b6acb21"
server_key_iv: "6477cc2f0c65277a16ea6c52a4472f07"
server_key_version: "encryptionkey@0"
I20260927 23:04:11.427644 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "1162e642335f4a26b9521c5f786ceeb3"
format_stamp: "Formatted at 2026-09-27 23:04:11 on dist-test-slave-lmfp"
server_key: "5cb34122e55294c3aa1c7da38b6acb21"
server_key_iv: "6477cc2f0c65277a16ea6c52a4472f07"
server_key_version: "encryptionkey@0"
I20260927 23:04:11.432703 7166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:04:11.436158 7181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:11.437397 7166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:04:11.437569 7166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "1162e642335f4a26b9521c5f786ceeb3"
format_stamp: "Formatted at 2026-09-27 23:04:11 on dist-test-slave-lmfp"
server_key: "5cb34122e55294c3aa1c7da38b6acb21"
server_key_iv: "6477cc2f0c65277a16ea6c52a4472f07"
server_key_version: "encryptionkey@0"
I20260927 23:04:11.437738 7166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:11.466799 7166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:11.471235 7166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:11.471506 7166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:11.472401 7166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:11.473680 7166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:11.473747 7166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:11.473903 7166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:11.473970 7166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:11.506060 7166 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:40101
I20260927 23:04:11.506105 7293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:40101 every 8 connection(s)
I20260927 23:04:11.507396 7166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:11.517004 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 7166
I20260927 23:04:11.517210 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:11.517911 4035 external_mini_cluster.cc:1428] Setting key 76996b08cf78bee980365789a140e10b
Sep 27 23:04:11 dist-test-slave-lmfp krb5kdc[7060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:11.526680 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:11.509800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:48617 (local address 127.3.240.254:35917)
0927 23:04:11.510221 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:11.510225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:11.511294 (+ 1069us) server_negotiation.cc:407] Connection header received
0927 23:04:11.512990 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:11.512995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:11.513083 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:11.513194 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:11.514551 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:11.515335 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:11.516891 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:11.517503 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:11.517634 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:11.521260 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:11.521266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:11.521270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:11.521310 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:11.523765 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:11.524462 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:11.524467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:11.524471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:11.524590 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:11.525027 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:11.525032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:11.525036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:11.525326 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:11.525477 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:11.526119 (+ 642us) server_negotiation.cc:299] Negotiation successful
0927 23:04:11.526323 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":226,"threads_started":1}
I20260927 23:04:11.527585 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:11.510453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35917 (local address 127.3.240.193:48617)
0927 23:04:11.511152 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:11.511187 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:11.512716 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:11.513412 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:11.513419 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:11.513872 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:11.514383 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:11.514398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:11.515494 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:11.515499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:11.516674 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:11.516713 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:11.517707 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:11.517714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:11.517929 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:11.518813 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:11.518832 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:11.521067 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:11.523898 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:11.523907 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:11.523921 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:11.524340 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:11.524727 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:11.524732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:11.524735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:11.524909 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:11.525450 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:11.525457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:11.525771 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 23:04:11.526024 (+ 253us) client_negotiation.cc:240] Negotiation successful
0927 23:04:11.526480 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":268,"threads_started":1}
I20260927 23:04:11.529035 7294 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35917
I20260927 23:04:11.529311 7294 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:11.529990 7294 heartbeater.cc:511] Master 127.3.240.254:35917 requested a full tablet report, sending...
I20260927 23:04:11.532289 7105 ts_manager.cc:194] Registered new tserver with Master: 1162e642335f4a26b9521c5f786ceeb3 (127.3.240.193:40101)
I20260927 23:04:11.534468 7105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:48617
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:04:11.557701 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35917
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:11.862835 7300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:11.863144 7300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:11.863237 7300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:11.870554 7300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:11.870661 7300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:11.870767 7300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:11.880716 7300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:11.882066 7300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:11.883428 7300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:11.896406 7305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:11.896436 7306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:11.896479 7308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:11.897202 7300 server_base.cc:1034] running on GCE node
I20260927 23:04:11.897792 7300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:11.898631 7300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:11.899917 7300 hybrid_clock.cc:648] HybridClock initialized: now 1790550251899847 us; error 46 us; skew 500 ppm
I20260927 23:04:11.900425 7300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:11 dist-test-slave-lmfp krb5kdc[7060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:11.903993 7300 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:04:11.905495 7300 webserver.cc:460] Webserver started at http://127.3.240.194:41129/ using document root <none> and password file <none>
I20260927 23:04:11.906221 7300 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:11.906337 7300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:11.906618 7300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:11.909770 7300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "9189f401680940319a8cd4a56edfa858"
format_stamp: "Formatted at 2026-09-27 23:04:11 on dist-test-slave-lmfp"
server_key: "47d5a7e228f08f6fb74df35628f2e8a4"
server_key_iv: "a2091a66e072ca56403a8b90c16a360f"
server_key_version: "encryptionkey@0"
I20260927 23:04:11.910597 7300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "9189f401680940319a8cd4a56edfa858"
format_stamp: "Formatted at 2026-09-27 23:04:11 on dist-test-slave-lmfp"
server_key: "47d5a7e228f08f6fb74df35628f2e8a4"
server_key_iv: "a2091a66e072ca56403a8b90c16a360f"
server_key_version: "encryptionkey@0"
I20260927 23:04:11.916047 7300 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 23:04:11.919716 7315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:11.921105 7300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:04:11.921241 7300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "9189f401680940319a8cd4a56edfa858"
format_stamp: "Formatted at 2026-09-27 23:04:11 on dist-test-slave-lmfp"
server_key: "47d5a7e228f08f6fb74df35628f2e8a4"
server_key_iv: "a2091a66e072ca56403a8b90c16a360f"
server_key_version: "encryptionkey@0"
I20260927 23:04:11.921415 7300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:11.934715 7300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:11.938380 7300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:11.938678 7300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:11.939594 7300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:11.940853 7300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:11.940948 7300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:11.941094 7300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:11.941150 7300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:11.971812 7300 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:39187
I20260927 23:04:11.971844 7427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:39187 every 8 connection(s)
I20260927 23:04:11.973009 7300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:11.983273 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 7300
I20260927 23:04:11.983464 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:11.984033 4035 external_mini_cluster.cc:1428] Setting key 6dff8dc802daa5459d67d97c02d8c28e
Sep 27 23:04:11 dist-test-slave-lmfp krb5kdc[7060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550251, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:11.992121 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:11.975510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:45997 (local address 127.3.240.254:35917)
0927 23:04:11.975700 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:11.975706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:11.976932 (+ 1226us) server_negotiation.cc:407] Connection header received
0927 23:04:11.978494 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:11.978498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:11.978557 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:11.978638 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:11.980008 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:11.980777 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:11.982149 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:11.982514 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:11.982599 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:11.986516 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:11.986521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:11.986524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:11.986551 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:11.989038 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:11.989874 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:11.989882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:11.989887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:11.990064 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:11.990523 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:11.990529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:11.990533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:11.990903 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:11.991108 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:11.991730 (+ 622us) server_negotiation.cc:299] Negotiation successful
0927 23:04:11.991882 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:04:11.993361 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:11.976072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35917 (local address 127.3.240.194:45997)
0927 23:04:11.976795 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:11.976830 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:11.978265 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:11.978818 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:11.978824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:11.979331 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:11.979843 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:11.979858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:11.980901 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:11.980904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:11.982032 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:11.982042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:11.982697 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:11.982701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:11.982941 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:11.983744 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:11.983771 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:11.986314 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:11.989166 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:11.989176 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:11.989193 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:11.989743 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:11.990183 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:11.990189 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:11.990194 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:11.990382 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:11.991099 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:11.991105 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:11.991365 (+ 260us) client_negotiation.cc:769] Sending connection context
0927 23:04:11.991626 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 23:04:11.992144 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":281,"threads_started":1}
I20260927 23:04:11.994665 7428 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35917
I20260927 23:04:11.994936 7428 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:11.995602 7428 heartbeater.cc:511] Master 127.3.240.254:35917 requested a full tablet report, sending...
I20260927 23:04:11.996892 7105 ts_manager.cc:194] Registered new tserver with Master: 9189f401680940319a8cd4a56edfa858 (127.3.240.194:39187)
I20260927 23:04:11.997695 7105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:45997
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:04:12.022020 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35917
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:12.327674 7433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:12.328028 7433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:12.328197 7433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:12.335500 7433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:12.335652 7433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:12.335834 7433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:12.345729 7433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:12.347114 7433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:12.348477 7433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:12.361325 7439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:12.361420 7441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:12.361454 7438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:12.362592 7433 server_base.cc:1034] running on GCE node
I20260927 23:04:12.363236 7433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:12.364089 7433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:12.365305 7433 hybrid_clock.cc:648] HybridClock initialized: now 1790550252365264 us; error 60 us; skew 500 ppm
I20260927 23:04:12.365595 7433 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550252, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:12.373731 7433 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:04:12.375233 7433 webserver.cc:460] Webserver started at http://127.3.240.195:43951/ using document root <none> and password file <none>
I20260927 23:04:12.375918 7433 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:12.376057 7433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:12.376351 7433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:12.379141 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "1f054c1cf2bd43f0abc6e4f288f72a40"
format_stamp: "Formatted at 2026-09-27 23:04:12 on dist-test-slave-lmfp"
server_key: "a4b46012970d7f0dfccc2e42f5d957ca"
server_key_iv: "bda424f86939465cba4e410a3929c845"
server_key_version: "encryptionkey@0"
I20260927 23:04:12.379851 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "1f054c1cf2bd43f0abc6e4f288f72a40"
format_stamp: "Formatted at 2026-09-27 23:04:12 on dist-test-slave-lmfp"
server_key: "a4b46012970d7f0dfccc2e42f5d957ca"
server_key_iv: "bda424f86939465cba4e410a3929c845"
server_key_version: "encryptionkey@0"
I20260927 23:04:12.384567 7433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:04:12.387952 7448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:12.389067 7433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:12.389220 7433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "1f054c1cf2bd43f0abc6e4f288f72a40"
format_stamp: "Formatted at 2026-09-27 23:04:12 on dist-test-slave-lmfp"
server_key: "a4b46012970d7f0dfccc2e42f5d957ca"
server_key_iv: "bda424f86939465cba4e410a3929c845"
server_key_version: "encryptionkey@0"
I20260927 23:04:12.389400 7433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:12.400148 7433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:12.404383 7433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:12.404665 7433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:12.405509 7433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:12.406755 7433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:12.406857 7433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:12.407021 7433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:12.407096 7433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 23:04:12.438796 7433 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:45175
I20260927 23:04:12.438834 7560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:45175 every 8 connection(s)
I20260927 23:04:12.440038 7433 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:12.446750 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 7433
I20260927 23:04:12.446913 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:12.447470 4035 external_mini_cluster.cc:1428] Setting key 8e9e4a38bd275527d6e60468dff37de0
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550252, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:12.458437 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:12.442525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:42965 (local address 127.3.240.254:35917)
0927 23:04:12.442696 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:12.442700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:12.443906 (+ 1206us) server_negotiation.cc:407] Connection header received
0927 23:04:12.445565 (+ 1659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:12.445570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:12.445651 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:12.445749 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:12.447467 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.448383 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.450286 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.450665 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.450783 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:12.453815 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:12.453819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:12.453823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:12.453852 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:12.456243 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.456806 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.456810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.456812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.456913 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.457225 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.457227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.457229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.457457 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:12.457596 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:12.458078 (+ 482us) server_negotiation.cc:299] Negotiation successful
0927 23:04:12.458224 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:04:12.459347 7563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:12.443080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35917 (local address 127.3.240.195:42965)
0927 23:04:12.443758 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:12.443799 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:12.445330 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:12.445993 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:12.446000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:12.446540 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:12.447289 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:12.447310 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.448543 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.448547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:12.450150 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:12.450164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.450855 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.450862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:12.451153 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:12.451686 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:12.451706 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:12.453653 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:12.456347 (+ 2694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:12.456353 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:12.456363 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:12.456709 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:12.456994 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:12.456996 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:12.456998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:12.457139 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:12.457580 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:12.457585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:12.457818 (+ 233us) client_negotiation.cc:769] Sending connection context
0927 23:04:12.458000 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 23:04:12.458292 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":261,"threads_started":1}
I20260927 23:04:12.460650 7561 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35917
I20260927 23:04:12.460944 7561 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:12.461776 7561 heartbeater.cc:511] Master 127.3.240.254:35917 requested a full tablet report, sending...
I20260927 23:04:12.463145 7105 ts_manager.cc:194] Registered new tserver with Master: 1f054c1cf2bd43f0abc6e4f288f72a40 (127.3.240.195:45175)
I20260927 23:04:12.463869 7105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:42965
I20260927 23:04:12.474189 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:12.518484 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:12.504613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.3.240.254:35917)
0927 23:04:12.504861 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:12.504868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:12.505163 (+ 295us) server_negotiation.cc:407] Connection header received
0927 23:04:12.505383 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:12.505387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:12.505459 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:12.505580 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:12.506940 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.507729 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.508967 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.509397 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.509534 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:12.513347 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:12.513352 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:12.513355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:12.513388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:12.515964 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.516568 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.516575 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.516578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.516707 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.517091 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.517097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.517100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.517406 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:12.517655 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:12.517956 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 23:04:12.518168 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260927 23:04:12.519105 7105 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60764
I20260927 23:04:12.520767 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7166
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:04:12.535831 7572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:12.522115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52978 (local address 127.3.240.193:40101)
0927 23:04:12.522721 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:12.522728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:12.522749 (+ 21us) server_negotiation.cc:407] Connection header received
0927 23:04:12.522799 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:12.522805 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:12.523059 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:12.523224 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:12.524351 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.525313 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.526502 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.527143 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.527273 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:12.530867 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:12.530878 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:12.530891 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:12.530921 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:12.533431 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.533975 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.533980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.533982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.534099 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.534390 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.534393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.534394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.534679 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:12.534874 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:12.535275 (+ 401us) server_negotiation.cc:299] Negotiation successful
0927 23:04:12.535453 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":241,"threads_started":1}
W20260927 23:04:12.537025 7289 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52978
I20260927 23:04:12.537400 7294 heartbeater.cc:503] Master 127.3.240.254:35917 was elected leader, sending a full tablet report...
W20260927 23:04:12.538234 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:04:12.538422 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 7166
W20260927 23:04:12.553426 7090 connection.cc:535] server connection from 127.3.240.193:48617 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 23:04:12.553735 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7300
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:04:12.568025 7573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:12.554539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.3.240.194:39187)
0927 23:04:12.555121 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:12.555128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:12.555153 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:04:12.555208 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:12.555214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:12.555415 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:12.555537 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:12.556698 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.557644 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.558846 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.559437 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.559538 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:12.563240 (+ 3702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:12.563250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:12.563264 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:12.563291 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:12.565856 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.566399 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.566406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.566408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.566514 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.566787 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.566790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.566792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.567152 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:12.567365 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:12.567580 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 23:04:12.567746 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":311,"threads_started":1}
W20260927 23:04:12.569228 7423 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49958
W20260927 23:04:12.570250 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7300: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:04:12.570396 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 7300
I20260927 23:04:12.585716 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7433
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550252, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:04:12.600776 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:12.586537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.3.240.195:45175)
0927 23:04:12.587149 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:12.587155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:12.587181 (+ 26us) server_negotiation.cc:407] Connection header received
0927 23:04:12.587231 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:12.587236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:12.587423 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:12.587604 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:12.588794 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.589745 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.590830 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:12.591446 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:12.591590 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:12.595761 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:12.595770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:12.595782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:12.595808 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:12.598263 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.598806 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.598811 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.598813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.598933 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:12.599438 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:12.599441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:12.599443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:12.599823 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:12.600030 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:12.600276 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 23:04:12.600448 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":285,"threads_started":1}
W20260927 23:04:12.601893 7556 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56076
W20260927 23:04:12.602885 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7433: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:04:12.603084 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 7433
I20260927 23:04:12.617581 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7073
I20260927 23:04:12.618435 7135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:12.715994 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 7073
2026-09-27T23:04:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3398 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7578](info): set up 2 sockets
Sep 27 23:04:12 dist-test-slave-lmfp krb5kdc[7578](info): commencing operation
krb5kdc: starting...
W20260927 23:04:13.751340 4035 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:13 dist-test-slave-lmfp krb5kdc[7578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:04:13.922628 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44421
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:14.231451 7591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:14.231729 7591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:14.231875 7591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:14.238517 7591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:14.238597 7591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:14.238687 7591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:14.238767 7591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:14.238821 7591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:14.247591 7591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:14.248736 7591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:14.249964 7591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:14.261344 7599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:14.261358 7597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:14.261209 7596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:14.262352 7591 server_base.cc:1034] running on GCE node
I20260927 23:04:14.263311 7591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:14.264890 7591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:14.266142 7591 hybrid_clock.cc:648] HybridClock initialized: now 1790550254266093 us; error 34 us; skew 500 ppm
I20260927 23:04:14.266496 7591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:14 dist-test-slave-lmfp krb5kdc[7578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:14.270421 7591 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:04:14.271864 7591 webserver.cc:460] Webserver started at http://127.3.240.254:40639/ using document root <none> and password file <none>
I20260927 23:04:14.272491 7591 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:14.272614 7591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:14.272917 7591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:14.275614 7591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "50732bb133514b0b9e05162625d2db33"
format_stamp: "Formatted at 2026-09-27 23:04:14 on dist-test-slave-lmfp"
server_key: "45e29e49a93565d710c5e776b7b8750f"
server_key_iv: "632d3d8aaf0bc18d6a3bf4b595b832dd"
server_key_version: "encryptionkey@0"
I20260927 23:04:14.276341 7591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "50732bb133514b0b9e05162625d2db33"
format_stamp: "Formatted at 2026-09-27 23:04:14 on dist-test-slave-lmfp"
server_key: "45e29e49a93565d710c5e776b7b8750f"
server_key_iv: "632d3d8aaf0bc18d6a3bf4b595b832dd"
server_key_version: "encryptionkey@0"
I20260927 23:04:14.280804 7591 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 23:04:14.283882 7606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:14.285058 7591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:04:14.285220 7591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "50732bb133514b0b9e05162625d2db33"
format_stamp: "Formatted at 2026-09-27 23:04:14 on dist-test-slave-lmfp"
server_key: "45e29e49a93565d710c5e776b7b8750f"
server_key_iv: "632d3d8aaf0bc18d6a3bf4b595b832dd"
server_key_version: "encryptionkey@0"
I20260927 23:04:14.285386 7591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:14.312734 7591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:14.316244 7591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:14.316504 7591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:14.333691 7591 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:44421
I20260927 23:04:14.333753 7657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:44421 every 8 connection(s)
I20260927 23:04:14.334872 7591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:14.338260 7658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:14.340481 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 7591
I20260927 23:04:14.340698 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:14.341245 4035 external_mini_cluster.cc:1428] Setting key 6fc8b463831f4ffd3aefcd5c9d925f25
I20260927 23:04:14.348944 7658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33: Bootstrap starting.
I20260927 23:04:14.352130 7658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:14.353319 7658 log.cc:824] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33: Log is configured to *not* fsync() on all Append() calls
Sep 27 23:04:14 dist-test-slave-lmfp krb5kdc[7578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:14.355897 7658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33: No bootstrap required, opened a new log
I20260927 23:04:14.361986 7661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:14.343029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51276 (local address 127.3.240.254:44421)
0927 23:04:14.343710 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:14.343720 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:14.343756 (+ 36us) server_negotiation.cc:407] Connection header received
0927 23:04:14.345197 (+ 1441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:14.345233 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:14.345573 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:14.345860 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:14.347544 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:14.348720 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:14.350376 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:14.350913 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:14.351098 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:14.355496 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:14.355503 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:14.355515 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:14.355543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:14.358486 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:14.359221 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:14.359226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:14.359231 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:14.359375 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:14.359730 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:14.359733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:14.359737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:14.360166 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:14.360461 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:14.360775 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 23:04:14.361109 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":306,"threads_started":1}
I20260927 23:04:14.363267 7658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50732bb133514b0b9e05162625d2db33" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44421 } }
I20260927 23:04:14.363669 7658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:14.363767 7658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50732bb133514b0b9e05162625d2db33, State: Initialized, Role: FOLLOWER
I20260927 23:04:14.364331 7658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50732bb133514b0b9e05162625d2db33" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44421 } }
I20260927 23:04:14.364601 7658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:14.364723 7658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:14.364825 7658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:14.366580 7658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50732bb133514b0b9e05162625d2db33" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44421 } }
I20260927 23:04:14.366952 7658 leader_election.cc:302] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50732bb133514b0b9e05162625d2db33; no voters:
I20260927 23:04:14.367478 7658 leader_election.cc:288] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:14.367602 7663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:14.368243 7663 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [term 1 LEADER]: Becoming Leader. State: Replica: 50732bb133514b0b9e05162625d2db33, State: Running, Role: LEADER
I20260927 23:04:14.368739 7663 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50732bb133514b0b9e05162625d2db33" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44421 } }
I20260927 23:04:14.369346 7658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:14.371059 7665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50732bb133514b0b9e05162625d2db33. Latest consensus state: current_term: 1 leader_uuid: "50732bb133514b0b9e05162625d2db33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50732bb133514b0b9e05162625d2db33" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44421 } } }
I20260927 23:04:14.371533 7665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:14.371315 7664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50732bb133514b0b9e05162625d2db33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50732bb133514b0b9e05162625d2db33" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44421 } } }
I20260927 23:04:14.371677 7664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33 [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:14.372282 7671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:14.376890 7671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:14.386243 7671 catalog_manager.cc:1347] Generated new cluster ID: 79b08c241e984775b7b7e949ba3bce34
I20260927 23:04:14.386346 7671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:14.410275 7671 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:14.411648 7671 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:14.419661 7671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50732bb133514b0b9e05162625d2db33: Generated new TSK 0
I20260927 23:04:14.420550 7671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:04:14.470696 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44421
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:14.793862 7684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:14.794163 7684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:14.794266 7684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:14.801753 7684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:14.801867 7684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:14.801987 7684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:14.811721 7684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:14.812970 7684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:14.814247 7684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:14.827421 7690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:14.827553 7692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:14.827523 7689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:14.828516 7684 server_base.cc:1034] running on GCE node
I20260927 23:04:14.829171 7684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:14.830112 7684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:14.831350 7684 hybrid_clock.cc:648] HybridClock initialized: now 1790550254831299 us; error 40 us; skew 500 ppm
I20260927 23:04:14.831660 7684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:14 dist-test-slave-lmfp krb5kdc[7578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:14.834585 7684 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:04:14.835974 7684 webserver.cc:460] Webserver started at http://127.3.240.193:44649/ using document root <none> and password file <none>
I20260927 23:04:14.836567 7684 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:14.836664 7684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:14.836946 7684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:14.839699 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "590b48248cff408883ce3683717a11eb"
format_stamp: "Formatted at 2026-09-27 23:04:14 on dist-test-slave-lmfp"
server_key: "9e9c24b7bba292f3792df7df0fefdd33"
server_key_iv: "220b3e8134c79e602f3102c198354e7e"
server_key_version: "encryptionkey@0"
I20260927 23:04:14.840509 7684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "590b48248cff408883ce3683717a11eb"
format_stamp: "Formatted at 2026-09-27 23:04:14 on dist-test-slave-lmfp"
server_key: "9e9c24b7bba292f3792df7df0fefdd33"
server_key_iv: "220b3e8134c79e602f3102c198354e7e"
server_key_version: "encryptionkey@0"
I20260927 23:04:14.845443 7684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:04:14.848661 7699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:14.849834 7684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:04:14.849998 7684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "590b48248cff408883ce3683717a11eb"
format_stamp: "Formatted at 2026-09-27 23:04:14 on dist-test-slave-lmfp"
server_key: "9e9c24b7bba292f3792df7df0fefdd33"
server_key_iv: "220b3e8134c79e602f3102c198354e7e"
server_key_version: "encryptionkey@0"
I20260927 23:04:14.850198 7684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:14.892365 7684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:14.896297 7684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:14.896586 7684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:14.897475 7684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:14.898701 7684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:14.898770 7684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:14.898926 7684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:14.899025 7684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:14.929627 7684 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:43247
I20260927 23:04:14.929692 7811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:43247 every 8 connection(s)
I20260927 23:04:14.930750 7684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:14.939492 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 7684
I20260927 23:04:14.939682 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:14.940377 4035 external_mini_cluster.cc:1428] Setting key b4b60e9d9188b8d95307ddf525c5f719
Sep 27 23:04:14 dist-test-slave-lmfp krb5kdc[7578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550254, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:14.949186 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:14.933057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:47101 (local address 127.3.240.254:44421)
0927 23:04:14.933422 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:14.933426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:14.934324 (+ 898us) server_negotiation.cc:407] Connection header received
0927 23:04:14.936155 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:14.936161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:14.936241 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:14.936348 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:14.937611 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:14.938317 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:14.940281 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:14.940698 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:14.940795 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:14.944155 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:14.944160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:14.944163 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:14.944191 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:14.946496 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:14.947226 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:14.947232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:14.947234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:14.947334 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:14.947718 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:14.947722 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:14.947725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:14.947937 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:14.948080 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:14.948675 (+ 595us) server_negotiation.cc:299] Negotiation successful
0927 23:04:14.948860 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":187,"threads_started":1}
I20260927 23:04:14.950013 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:14.933587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44421 (local address 127.3.240.193:47101)
0927 23:04:14.934190 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:14.934229 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:14.935899 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:14.936567 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:14.936574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:14.936987 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:14.937457 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:14.937481 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:14.938460 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:14.938470 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:14.940122 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:14.940137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:14.940902 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:14.940908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:14.941082 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:14.941663 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:14.941688 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:14.943954 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:14.946622 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:14.946629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:14.946643 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:14.947100 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:14.947439 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:14.947443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:14.947446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:14.947612 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:14.948055 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:14.948061 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:14.948345 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 23:04:14.948568 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 23:04:14.948989 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":237,"threads_started":1}
I20260927 23:04:14.951421 7812 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44421
I20260927 23:04:14.951679 7812 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:14.952322 7812 heartbeater.cc:511] Master 127.3.240.254:44421 requested a full tablet report, sending...
I20260927 23:04:14.954525 7623 ts_manager.cc:194] Registered new tserver with Master: 590b48248cff408883ce3683717a11eb (127.3.240.193:43247)
I20260927 23:04:14.956557 7623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:47101
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:04:14.979077 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44421
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:15.279199 7818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:15.279531 7818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:15.279708 7818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:15.286530 7818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:15.286665 7818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:15.286834 7818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:15.296557 7818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:15.297854 7818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:15.299155 7818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:15.311843 7823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:15.312019 7826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:15.311843 7824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:15.312553 7818 server_base.cc:1034] running on GCE node
I20260927 23:04:15.313130 7818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:15.313988 7818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:15.315196 7818 hybrid_clock.cc:648] HybridClock initialized: now 1790550255315164 us; error 46 us; skew 500 ppm
I20260927 23:04:15.315474 7818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:15 dist-test-slave-lmfp krb5kdc[7578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:15.318353 7818 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:04:15.319694 7818 webserver.cc:460] Webserver started at http://127.3.240.194:34919/ using document root <none> and password file <none>
I20260927 23:04:15.320353 7818 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:15.320484 7818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:15.320765 7818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:15.323405 7818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "97249277e33143109d32895f8d1bdeec"
format_stamp: "Formatted at 2026-09-27 23:04:15 on dist-test-slave-lmfp"
server_key: "3cd15aee7fb46b953d4f6b27c017efae"
server_key_iv: "d3c6ff4c68320e623e0462b9e9454fcf"
server_key_version: "encryptionkey@0"
I20260927 23:04:15.324133 7818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "97249277e33143109d32895f8d1bdeec"
format_stamp: "Formatted at 2026-09-27 23:04:15 on dist-test-slave-lmfp"
server_key: "3cd15aee7fb46b953d4f6b27c017efae"
server_key_iv: "d3c6ff4c68320e623e0462b9e9454fcf"
server_key_version: "encryptionkey@0"
I20260927 23:04:15.328884 7818 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 23:04:15.331750 7833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:15.333190 7818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:04:15.333343 7818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "97249277e33143109d32895f8d1bdeec"
format_stamp: "Formatted at 2026-09-27 23:04:15 on dist-test-slave-lmfp"
server_key: "3cd15aee7fb46b953d4f6b27c017efae"
server_key_iv: "d3c6ff4c68320e623e0462b9e9454fcf"
server_key_version: "encryptionkey@0"
I20260927 23:04:15.333539 7818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:15.356642 7818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:15.360174 7818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:15.360426 7818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:15.361238 7818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:15.362447 7818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:15.362552 7818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:15.362702 7818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:15.362774 7818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:15.393693 7818 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:33883
I20260927 23:04:15.393764 7945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:33883 every 8 connection(s)
I20260927 23:04:15.394822 7818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:15.403792 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 7818
I20260927 23:04:15.403990 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:15.404349 4035 external_mini_cluster.cc:1428] Setting key 16fb70c4559e41bf1765410dea3dc584
Sep 27 23:04:15 dist-test-slave-lmfp krb5kdc[7578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:15.413434 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:15.397407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:44457 (local address 127.3.240.254:44421)
0927 23:04:15.397585 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:15.397589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:15.398610 (+ 1021us) server_negotiation.cc:407] Connection header received
0927 23:04:15.400412 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:15.400415 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:15.400472 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:15.400562 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:15.401818 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.402476 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.403976 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.404408 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.404571 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:15.408056 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:15.408061 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:15.408064 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:15.408094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:15.410577 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.411346 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.411355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.411358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.411477 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.411883 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.411887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.411892 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.412154 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:15.412304 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:15.412922 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 23:04:15.413133 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:04:15.414287 7948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:15.397836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44421 (local address 127.3.240.194:44457)
0927 23:04:15.398466 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:15.398501 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:15.400195 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:15.400735 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:15.400742 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:15.401179 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:15.401663 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:15.401677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.402608 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.402611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:15.403823 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:15.403834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.404632 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.404639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:15.404927 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:15.405594 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:15.405613 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:15.407838 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:15.410708 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:15.410728 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:15.410743 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:15.411223 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:15.411593 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:15.411597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:15.411603 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:15.411766 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:15.412283 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:15.412290 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:15.412573 (+ 283us) client_negotiation.cc:769] Sending connection context
0927 23:04:15.412829 (+ 256us) client_negotiation.cc:240] Negotiation successful
0927 23:04:15.413291 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":253,"threads_started":1}
I20260927 23:04:15.415591 7946 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44421
I20260927 23:04:15.415894 7946 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:15.416536 7946 heartbeater.cc:511] Master 127.3.240.254:44421 requested a full tablet report, sending...
I20260927 23:04:15.417894 7623 ts_manager.cc:194] Registered new tserver with Master: 97249277e33143109d32895f8d1bdeec (127.3.240.194:33883)
I20260927 23:04:15.419158 7623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:44457
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:04:15.442068 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44421
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:15.741417 7951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:15.741663 7951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:15.741808 7951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:15.748749 7951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:15.748857 7951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:15.749008 7951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:15.758059 7951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:15.759202 7951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:15.760389 7951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:15.772078 7956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:15.772101 7957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:15.772135 7959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:15.773324 7951 server_base.cc:1034] running on GCE node
I20260927 23:04:15.773803 7951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:15.774590 7951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:15.775766 7951 hybrid_clock.cc:648] HybridClock initialized: now 1790550255775711 us; error 48 us; skew 500 ppm
I20260927 23:04:15.776005 7951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:15 dist-test-slave-lmfp krb5kdc[7578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:15.778736 7951 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:04:15.780197 7951 webserver.cc:460] Webserver started at http://127.3.240.195:33255/ using document root <none> and password file <none>
I20260927 23:04:15.780810 7951 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:15.780953 7951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:15.781265 7951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:15.783927 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "47f01d41529a4662857f0f6bf9bc1c55"
format_stamp: "Formatted at 2026-09-27 23:04:15 on dist-test-slave-lmfp"
server_key: "9ede7e3511d5bcbe07795f846a44fa51"
server_key_iv: "fe5bca874883e40d64264b7bbf79a241"
server_key_version: "encryptionkey@0"
I20260927 23:04:15.784668 7951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "47f01d41529a4662857f0f6bf9bc1c55"
format_stamp: "Formatted at 2026-09-27 23:04:15 on dist-test-slave-lmfp"
server_key: "9ede7e3511d5bcbe07795f846a44fa51"
server_key_iv: "fe5bca874883e40d64264b7bbf79a241"
server_key_version: "encryptionkey@0"
I20260927 23:04:15.789337 7951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:04:15.792414 7966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:15.793582 7951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 23:04:15.793735 7951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "47f01d41529a4662857f0f6bf9bc1c55"
format_stamp: "Formatted at 2026-09-27 23:04:15 on dist-test-slave-lmfp"
server_key: "9ede7e3511d5bcbe07795f846a44fa51"
server_key_iv: "fe5bca874883e40d64264b7bbf79a241"
server_key_version: "encryptionkey@0"
I20260927 23:04:15.793906 7951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:15.811906 7951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:15.815539 7951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:15.815791 7951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:15.816612 7951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:15.817744 7951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:15.817835 7951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:15.817986 7951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:15.818049 7951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:15.850175 7951 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:34261
I20260927 23:04:15.850220 8078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:34261 every 8 connection(s)
I20260927 23:04:15.851428 7951 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:15.857358 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 7951
I20260927 23:04:15.857556 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:15.858042 4035 external_mini_cluster.cc:1428] Setting key b4f4541f3bff96942d5375ae406ed07b
Sep 27 23:04:15 dist-test-slave-lmfp krb5kdc[7578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:15.871587 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:15.853852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47449 (local address 127.3.240.254:44421)
0927 23:04:15.854027 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:15.854031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:15.855264 (+ 1233us) server_negotiation.cc:407] Connection header received
0927 23:04:15.857026 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:15.857030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:15.857092 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:15.857173 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:15.859056 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.860009 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.861531 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.861959 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.862095 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:15.865505 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:15.865513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:15.865523 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:15.865565 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:15.868151 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.868978 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.868984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.868988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.869118 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.869565 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.869572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.869576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.869857 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:15.870236 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:15.870961 (+ 725us) server_negotiation.cc:299] Negotiation successful
0927 23:04:15.871229 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:04:15.872673 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:15.854382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44421 (local address 127.3.240.195:47449)
0927 23:04:15.855092 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:15.855129 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:15.856757 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:15.857479 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:15.857491 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:15.858141 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:15.858845 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:15.858879 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.860155 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.860162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:15.861375 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:15.861386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.862210 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.862220 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:15.862462 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:15.863204 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:15.863230 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:15.865269 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:15.868336 (+ 3067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:15.868345 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:15.868361 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:15.868832 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:15.869236 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:15.869242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:15.869246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:15.869446 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:15.870206 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:15.870214 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:15.870537 (+ 323us) client_negotiation.cc:769] Sending connection context
0927 23:04:15.870866 (+ 329us) client_negotiation.cc:240] Negotiation successful
0927 23:04:15.871404 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":244,"threads_started":1}
I20260927 23:04:15.873991 8079 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44421
I20260927 23:04:15.874276 8079 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:15.874961 8079 heartbeater.cc:511] Master 127.3.240.254:44421 requested a full tablet report, sending...
I20260927 23:04:15.876335 7623 ts_manager.cc:194] Registered new tserver with Master: 47f01d41529a4662857f0f6bf9bc1c55 (127.3.240.195:34261)
I20260927 23:04:15.877134 7623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:47449
I20260927 23:04:15.885269 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 27 23:04:15 dist-test-slave-lmfp krb5kdc[7578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 23:04:15 dist-test-slave-lmfp krb5kdc[7578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:04:15.939448 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:15.923492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.3.240.193:43247)
0927 23:04:15.924198 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:15.924205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:15.924230 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:04:15.924315 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:15.924321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:15.924536 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:15.924660 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:15.925995 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.927053 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.928555 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.929094 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.929187 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:15.934045 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:15.934055 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:15.934067 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:15.934094 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:15.936965 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.937657 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.937663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.937665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.937769 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.938140 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.938143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.938145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.938429 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:15.938631 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:15.938849 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 23:04:15.939091 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":475,"threads_started":1}
Sep 27 23:04:15 dist-test-slave-lmfp krb5kdc[7578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:15.955335 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:15.943218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51284 (local address 127.3.240.254:44421)
0927 23:04:15.943419 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:15.943424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:15.943588 (+ 164us) server_negotiation.cc:407] Connection header received
0927 23:04:15.943712 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:15.943716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:15.943769 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:15.943851 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:15.944976 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.945640 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.946948 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.947446 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.947543 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:15.950887 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:15.950891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:15.950894 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:15.950922 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:15.953162 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.953668 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.953671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.953673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.953783 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.954103 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.954107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.954109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.954370 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:15.954576 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:15.954748 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 23:04:15.954967 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 23:04:15.956099 7623 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51284
I20260927 23:04:15.956909 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7684
I20260927 23:04:15.959367 7812 heartbeater.cc:503] Master 127.3.240.254:44421 was elected leader, sending a full tablet report...
I20260927 23:04:15.968223 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:15.957639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.3.240.193:43247)
0927 23:04:15.957884 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:15.957890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:15.958081 (+ 191us) server_negotiation.cc:407] Connection header received
0927 23:04:15.958259 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:15.958264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:15.958343 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:15.958440 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:15.959737 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.960354 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.961550 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:15.961924 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:15.962052 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:15.964108 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:15.964113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:15.964118 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:15.964154 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:15.966269 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.966843 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.966847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.966850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.966960 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:15.967272 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:15.967274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:15.967276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:15.967472 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:15.967623 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:15.967784 (+ 161us) server_negotiation.cc:299] Negotiation successful
0927 23:04:15.967915 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:04:15.968688 7807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:16.047384 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 7684
I20260927 23:04:16.063705 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7818
Sep 27 23:04:16 dist-test-slave-lmfp krb5kdc[7578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:04:16.078104 8091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:16.064547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.3.240.194:33883)
0927 23:04:16.065105 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:16.065112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:16.065138 (+ 26us) server_negotiation.cc:407] Connection header received
0927 23:04:16.065196 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:16.065202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:16.065430 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:16.065559 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:16.066611 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:16.067676 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:16.068838 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:16.069337 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:16.069510 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:16.073291 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:16.073302 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:16.073316 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:16.073351 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:16.075954 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:16.076523 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:16.076530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:16.076533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:16.076635 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:16.076913 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:16.076916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:16.076918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:16.077218 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:16.077424 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:16.077656 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 23:04:16.077831 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":306,"threads_started":1}
I20260927 23:04:16.079352 7941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:16.158566 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 7818
I20260927 23:04:16.174227 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7951
Sep 27 23:04:16 dist-test-slave-lmfp krb5kdc[7578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:04:16.190481 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:16.175127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.3.240.195:34261)
0927 23:04:16.175764 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:16.175770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:16.175797 (+ 27us) server_negotiation.cc:407] Connection header received
0927 23:04:16.175854 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:16.175860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:16.176066 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:16.176193 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:16.177292 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:16.178329 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:16.180070 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:16.180725 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:16.180826 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:16.184922 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:16.184933 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:16.184947 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:16.184983 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:16.187959 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:16.188625 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:16.188632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:16.188634 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:16.188744 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:16.189192 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:16.189197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:16.189206 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:16.189520 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:16.189769 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:16.190019 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 23:04:16.190199 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":307,"threads_started":1}
I20260927 23:04:16.191766 8074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:16.271435 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 7951
I20260927 23:04:16.286883 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 7591
I20260927 23:04:16.288162 7653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:16.383687 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 7591
2026-09-27T23:04:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3666 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:16 dist-test-slave-lmfp krb5kdc[8098](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:16 dist-test-slave-lmfp krb5kdc[8098](info): set up 2 sockets
Sep 27 23:04:16 dist-test-slave-lmfp krb5kdc[8098](info): commencing operation
krb5kdc: starting...
W20260927 23:04:17.455418 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:17 dist-test-slave-lmfp krb5kdc[8098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:04:17.636925 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:33259
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:33259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:17.939082 8111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:17.939337 8111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:17.939431 8111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:17.946033 8111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:17.946132 8111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:17.946175 8111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:17.946229 8111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:17.946275 8111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:17.955503 8111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:33259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:17.956650 8111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:17.957854 8111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:17.968899 8119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:17.968933 8117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:17.968957 8116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:17.969980 8111 server_base.cc:1034] running on GCE node
I20260927 23:04:17.970708 8111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:17.972061 8111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:17.973255 8111 hybrid_clock.cc:648] HybridClock initialized: now 1790550257973222 us; error 28 us; skew 500 ppm
I20260927 23:04:17.973532 8111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:17 dist-test-slave-lmfp krb5kdc[8098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550257, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:17.979980 8111 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:04:17.980990 8111 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:17.981070 8111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:17.981312 8111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:17.984159 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "9598d0ae122441c4bd3cb1ab6209d64c"
format_stamp: "Formatted at 2026-09-27 23:04:17 on dist-test-slave-lmfp"
server_key: "d41b133aeab3aa0db5a3c411970e62d5"
server_key_iv: "63f60d72dcfe440947db331735d6feb8"
server_key_version: "encryptionkey@0"
I20260927 23:04:17.984807 8111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "9598d0ae122441c4bd3cb1ab6209d64c"
format_stamp: "Formatted at 2026-09-27 23:04:17 on dist-test-slave-lmfp"
server_key: "d41b133aeab3aa0db5a3c411970e62d5"
server_key_iv: "63f60d72dcfe440947db331735d6feb8"
server_key_version: "encryptionkey@0"
I20260927 23:04:17.989226 8111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 23:04:17.992264 8125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:17.993395 8111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260927 23:04:17.993563 8111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "9598d0ae122441c4bd3cb1ab6209d64c"
format_stamp: "Formatted at 2026-09-27 23:04:17 on dist-test-slave-lmfp"
server_key: "d41b133aeab3aa0db5a3c411970e62d5"
server_key_iv: "63f60d72dcfe440947db331735d6feb8"
server_key_version: "encryptionkey@0"
I20260927 23:04:17.993757 8111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:18.009462 8111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:18.013492 8111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:18.013716 8111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:18.030804 8111 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:33259
I20260927 23:04:18.030824 8176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:33259 every 8 connection(s)
I20260927 23:04:18.031647 8111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:18.032538 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 8111
I20260927 23:04:18.032779 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:18.033305 4035 external_mini_cluster.cc:1428] Setting key fe313910c09980279f89ee3bbd2448ff
I20260927 23:04:18.035730 8177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:18.047869 8177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c: Bootstrap starting.
Sep 27 23:04:18 dist-test-slave-lmfp krb5kdc[8098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:18.051144 8177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:18.052166 8177 log.cc:824] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:18.055197 8177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c: No bootstrap required, opened a new log
I20260927 23:04:18.055976 8180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:18.035130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40060 (local address 127.3.240.254:33259)
0927 23:04:18.035933 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:18.035947 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:18.036008 (+ 61us) server_negotiation.cc:407] Connection header received
0927 23:04:18.038222 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:18.038247 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:18.038745 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:18.039178 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:18.040577 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:18.042593 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:18.043977 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:18.045005 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:18.045212 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:18.049147 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:18.049154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:18.049166 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:18.049201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:18.052080 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:18.052911 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:18.052920 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:18.052927 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:18.053168 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:18.053569 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:18.053572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:18.053574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:18.054025 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:18.054234 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:18.054630 (+ 396us) server_negotiation.cc:299] Negotiation successful
0927 23:04:18.054945 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":323,"threads_started":1}
I20260927 23:04:18.062208 8177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9598d0ae122441c4bd3cb1ab6209d64c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33259 } }
I20260927 23:04:18.062525 8177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:18.062618 8177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9598d0ae122441c4bd3cb1ab6209d64c, State: Initialized, Role: FOLLOWER
I20260927 23:04:18.063208 8177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9598d0ae122441c4bd3cb1ab6209d64c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33259 } }
I20260927 23:04:18.063431 8177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:18.063563 8177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:18.063725 8177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:18.065907 8177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9598d0ae122441c4bd3cb1ab6209d64c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33259 } }
I20260927 23:04:18.066355 8177 leader_election.cc:302] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9598d0ae122441c4bd3cb1ab6209d64c; no voters:
I20260927 23:04:18.066951 8177 leader_election.cc:288] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:18.067111 8182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:18.068207 8182 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [term 1 LEADER]: Becoming Leader. State: Replica: 9598d0ae122441c4bd3cb1ab6209d64c, State: Running, Role: LEADER
I20260927 23:04:18.068686 8182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9598d0ae122441c4bd3cb1ab6209d64c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33259 } }
I20260927 23:04:18.068970 8177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:18.071424 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9598d0ae122441c4bd3cb1ab6209d64c. Latest consensus state: current_term: 1 leader_uuid: "9598d0ae122441c4bd3cb1ab6209d64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9598d0ae122441c4bd3cb1ab6209d64c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33259 } } }
I20260927 23:04:18.071861 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:18.071393 8183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9598d0ae122441c4bd3cb1ab6209d64c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9598d0ae122441c4bd3cb1ab6209d64c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33259 } } }
I20260927 23:04:18.072340 8183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:18.073624 8189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:18.077924 8189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:18.085955 8189 catalog_manager.cc:1347] Generated new cluster ID: be5e78fa95dd46aa914d10ecb6623660
I20260927 23:04:18.086050 8189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:18.104374 8189 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:18.105495 8189 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:18.119071 8189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c: Generated new TSK 0
I20260927 23:04:18.119877 8189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:04:18.163053 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33259
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:18.474216 8203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:18.474471 8203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:18.474565 8203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:18.481829 8203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:18.481918 8203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:18.482026 8203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:18.491031 8203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:18.492240 8203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:18.493439 8203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:18.505661 8208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:18.505688 8209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:18.505720 8211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:18.506364 8203 server_base.cc:1034] running on GCE node
I20260927 23:04:18.506966 8203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:18.507876 8203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:18.509100 8203 hybrid_clock.cc:648] HybridClock initialized: now 1790550258509054 us; error 41 us; skew 500 ppm
I20260927 23:04:18.509387 8203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:18 dist-test-slave-lmfp krb5kdc[8098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550258, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:18.512300 8203 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:04:18.513298 8203 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:18.513378 8203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:18.513613 8203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:18.516268 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "4609b5c8cf0f451e929b9e5195222f70"
format_stamp: "Formatted at 2026-09-27 23:04:18 on dist-test-slave-lmfp"
server_key: "3cccf1b4083796c22d571b487b184139"
server_key_iv: "097b54fa90024e645bd80ec6f000a35c"
server_key_version: "encryptionkey@0"
I20260927 23:04:18.517006 8203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "4609b5c8cf0f451e929b9e5195222f70"
format_stamp: "Formatted at 2026-09-27 23:04:18 on dist-test-slave-lmfp"
server_key: "3cccf1b4083796c22d571b487b184139"
server_key_iv: "097b54fa90024e645bd80ec6f000a35c"
server_key_version: "encryptionkey@0"
I20260927 23:04:18.522011 8203 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 23:04:18.525373 8217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:18.526701 8203 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260927 23:04:18.526885 8203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "4609b5c8cf0f451e929b9e5195222f70"
format_stamp: "Formatted at 2026-09-27 23:04:18 on dist-test-slave-lmfp"
server_key: "3cccf1b4083796c22d571b487b184139"
server_key_iv: "097b54fa90024e645bd80ec6f000a35c"
server_key_version: "encryptionkey@0"
I20260927 23:04:18.527161 8203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:18.552067 8203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:18.556368 8203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:18.556672 8203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:18.557510 8203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:18.558764 8203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:18.558887 8203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:18.559034 8203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:18.559128 8203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 23:04:18.590557 8203 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:39103
I20260927 23:04:18.590590 8329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:39103 every 8 connection(s)
I20260927 23:04:18.591439 8203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:18.600813 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 8203
I20260927 23:04:18.601063 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:18.601835 4035 external_mini_cluster.cc:1428] Setting key 16e6db9e221dbce8077d316251326b13
Sep 27 23:04:18 dist-test-slave-lmfp krb5kdc[8098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550258, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:18.611265 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:18.593705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:48287 (local address 127.3.240.254:33259)
0927 23:04:18.594218 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:18.594226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:18.595152 (+ 926us) server_negotiation.cc:407] Connection header received
0927 23:04:18.597117 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:18.597123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:18.597210 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:18.597345 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:18.598852 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:18.599641 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:18.601686 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:18.602106 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:18.602198 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:18.606031 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:18.606035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:18.606038 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:18.606065 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:18.608442 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:18.609247 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:18.609254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:18.609265 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:18.609368 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:18.609746 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:18.609751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:18.609755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:18.609987 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:18.610125 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:18.610716 (+ 591us) server_negotiation.cc:299] Negotiation successful
0927 23:04:18.610905 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":246,"threads_started":1}
I20260927 23:04:18.612171 8333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:18.594285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:33259 (local address 127.3.240.193:48287)
0927 23:04:18.594932 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:18.594973 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:18.596789 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:18.597574 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:18.597581 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:18.598128 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:18.598677 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:18.598694 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:18.599795 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:18.599802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:18.601544 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:18.601559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:18.602313 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:18.602320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:18.602592 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:18.603407 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:18.603431 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:18.605814 (+ 2383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:18.608573 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:18.608581 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:18.608598 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:18.609148 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:18.609477 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:18.609481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:18.609485 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:18.609645 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:18.610101 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:18.610108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:18.610406 (+ 298us) client_negotiation.cc:769] Sending connection context
0927 23:04:18.610643 (+ 237us) client_negotiation.cc:240] Negotiation successful
0927 23:04:18.611116 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":250,"threads_started":1}
I20260927 23:04:18.613648 8330 heartbeater.cc:347] Connected to a master server at 127.3.240.254:33259
I20260927 23:04:18.613937 8330 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:18.614560 8330 heartbeater.cc:511] Master 127.3.240.254:33259 requested a full tablet report, sending...
I20260927 23:04:18.616933 8142 ts_manager.cc:194] Registered new tserver with Master: 4609b5c8cf0f451e929b9e5195222f70 (127.3.240.193:39103)
I20260927 23:04:18.619187 8142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:48287
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:04:18.644019 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33259
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:18.950956 8336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:18.951236 8336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:18.951327 8336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:18.958303 8336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:18.958416 8336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:18.958540 8336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:18.967504 8336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:18.968590 8336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:18.969868 8336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:18.982012 8341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:18.982101 8342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:18.982034 8344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:18.983323 8336 server_base.cc:1034] running on GCE node
I20260927 23:04:18.983896 8336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:18.984658 8336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:18.985841 8336 hybrid_clock.cc:648] HybridClock initialized: now 1790550258985784 us; error 52 us; skew 500 ppm
I20260927 23:04:18.986132 8336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:18 dist-test-slave-lmfp krb5kdc[8098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550258, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:18.988960 8336 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:04:18.990011 8336 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:18.990136 8336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:18.990415 8336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:18.993045 8336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "e4e956de80e643f391c75d0330b9d9fa"
format_stamp: "Formatted at 2026-09-27 23:04:18 on dist-test-slave-lmfp"
server_key: "8319b06d397619d5a302cb0fdf279e64"
server_key_iv: "51e4aea19231417b6df8c16715471333"
server_key_version: "encryptionkey@0"
I20260927 23:04:18.993757 8336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "e4e956de80e643f391c75d0330b9d9fa"
format_stamp: "Formatted at 2026-09-27 23:04:18 on dist-test-slave-lmfp"
server_key: "8319b06d397619d5a302cb0fdf279e64"
server_key_iv: "51e4aea19231417b6df8c16715471333"
server_key_version: "encryptionkey@0"
I20260927 23:04:18.998704 8336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260927 23:04:19.001778 8350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:19.003096 8336 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:19.003253 8336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "e4e956de80e643f391c75d0330b9d9fa"
format_stamp: "Formatted at 2026-09-27 23:04:18 on dist-test-slave-lmfp"
server_key: "8319b06d397619d5a302cb0fdf279e64"
server_key_iv: "51e4aea19231417b6df8c16715471333"
server_key_version: "encryptionkey@0"
I20260927 23:04:19.003451 8336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:19.015831 8336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:19.019176 8336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:19.019416 8336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:19.020284 8336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:19.021489 8336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:19.021596 8336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:19.021721 8336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:19.021801 8336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:19.053056 8336 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:36755
I20260927 23:04:19.053134 8462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:36755 every 8 connection(s)
I20260927 23:04:19.053891 8336 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:19.058884 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 8336
I20260927 23:04:19.059087 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:19.059520 4035 external_mini_cluster.cc:1428] Setting key a9339a47135c33ff8928e125f50db44e
Sep 27 23:04:19 dist-test-slave-lmfp krb5kdc[8098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550258, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:19.074065 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.056134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:35405 (local address 127.3.240.254:33259)
0927 23:04:19.056320 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.056326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.057399 (+ 1073us) server_negotiation.cc:407] Connection header received
0927 23:04:19.058882 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.058887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.058970 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.059110 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:19.060667 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.061640 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.063961 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.064340 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.064442 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.068732 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:19.068738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:19.068748 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:19.068792 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:19.071393 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.072024 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.072032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.072034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.072144 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.072499 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.072505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.072510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.072749 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:19.072898 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.073505 (+ 607us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.073706 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":67}
I20260927 23:04:19.074919 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.056597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:33259 (local address 127.3.240.194:35405)
0927 23:04:19.057258 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.057290 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:19.058644 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:19.059332 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:19.059339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:19.059870 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:19.060482 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.060517 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.061771 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.061776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.063831 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.063847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.064564 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.064572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.064841 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.065521 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:19.065558 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:19.068511 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:19.071520 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:19.071528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:19.071541 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:19.071901 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:19.072240 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:19.072245 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:19.072248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:19.072400 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:19.072865 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:19.072872 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:19.073171 (+ 299us) client_negotiation.cc:769] Sending connection context
0927 23:04:19.073438 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 23:04:19.073922 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":289,"threads_started":1}
I20260927 23:04:19.076303 8463 heartbeater.cc:347] Connected to a master server at 127.3.240.254:33259
I20260927 23:04:19.076591 8463 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:19.077224 8463 heartbeater.cc:511] Master 127.3.240.254:33259 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260927 23:04:19.078581 8142 ts_manager.cc:194] Registered new tserver with Master: e4e956de80e643f391c75d0330b9d9fa (127.3.240.194:36755)
I20260927 23:04:19.079488 8142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:35405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:04:19.101378 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33259
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:19.410423 8468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:19.410658 8468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:19.410810 8468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:19.417722 8468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:19.417817 8468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:19.418046 8468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:19.426978 8468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33259
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:19.428210 8468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:19.429457 8468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:19.441124 8473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:19.441090 8476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:19.441159 8474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:19.441872 8468 server_base.cc:1034] running on GCE node
I20260927 23:04:19.442469 8468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:19.443424 8468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:19.444645 8468 hybrid_clock.cc:648] HybridClock initialized: now 1790550259444611 us; error 44 us; skew 500 ppm
I20260927 23:04:19.444938 8468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:19 dist-test-slave-lmfp krb5kdc[8098](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550259, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:19.447824 8468 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:04:19.448763 8468 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:19.448839 8468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:19.449093 8468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:19.451591 8468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "192d9a99420e49c3a9133e6cefbbead1"
format_stamp: "Formatted at 2026-09-27 23:04:19 on dist-test-slave-lmfp"
server_key: "cc1bf79e523f0c4db74262bae6ba9b92"
server_key_iv: "8c6597bc9d1f900d2135041e2141a6ad"
server_key_version: "encryptionkey@0"
I20260927 23:04:19.452244 8468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "192d9a99420e49c3a9133e6cefbbead1"
format_stamp: "Formatted at 2026-09-27 23:04:19 on dist-test-slave-lmfp"
server_key: "cc1bf79e523f0c4db74262bae6ba9b92"
server_key_iv: "8c6597bc9d1f900d2135041e2141a6ad"
server_key_version: "encryptionkey@0"
I20260927 23:04:19.456658 8468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 23:04:19.459517 8482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:19.460649 8468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:19.460790 8468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "192d9a99420e49c3a9133e6cefbbead1"
format_stamp: "Formatted at 2026-09-27 23:04:19 on dist-test-slave-lmfp"
server_key: "cc1bf79e523f0c4db74262bae6ba9b92"
server_key_iv: "8c6597bc9d1f900d2135041e2141a6ad"
server_key_version: "encryptionkey@0"
I20260927 23:04:19.460942 8468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:19.471678 8468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:19.475499 8468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:19.475751 8468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:19.476670 8468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:19.477870 8468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:19.477955 8468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:19.478080 8468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:19.478150 8468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:19.508780 8468 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:43315
I20260927 23:04:19.508829 8594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:43315 every 8 connection(s)
I20260927 23:04:19.509605 8468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:19.518859 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 8468
I20260927 23:04:19.519074 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:19.519490 4035 external_mini_cluster.cc:1428] Setting key e631ddb4781526679d684890cc90b1b8
Sep 27 23:04:19 dist-test-slave-lmfp krb5kdc[8098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550259, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:19.527179 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.511869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:40073 (local address 127.3.240.254:33259)
0927 23:04:19.512031 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.512036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.513033 (+ 997us) server_negotiation.cc:407] Connection header received
0927 23:04:19.514737 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.514742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.514838 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.514935 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:19.516322 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.517033 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.518403 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.518812 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.518942 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.522425 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:19.522429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:19.522432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:19.522459 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:19.524792 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.525373 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.525376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.525378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.525484 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.525830 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.525833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.525835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.526068 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:19.526172 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.526719 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.526867 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 23:04:19.527941 8597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.512268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:33259 (local address 127.3.240.195:40073)
0927 23:04:19.512892 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.512929 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:19.514475 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:19.515196 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:19.515204 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:19.515651 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:19.516176 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.516191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.517174 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.517184 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.518268 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.518278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.519049 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.519057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.519318 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.519937 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:19.519960 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:19.522261 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:19.524898 (+ 2637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:19.524903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:19.524913 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:19.525282 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:19.525570 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:19.525573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:19.525575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:19.525726 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:19.526167 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:19.526172 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:19.526433 (+ 261us) client_negotiation.cc:769] Sending connection context
0927 23:04:19.526651 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 23:04:19.526977 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":269,"threads_started":1}
I20260927 23:04:19.529141 8595 heartbeater.cc:347] Connected to a master server at 127.3.240.254:33259
I20260927 23:04:19.529383 8595 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:19.529933 8595 heartbeater.cc:511] Master 127.3.240.254:33259 requested a full tablet report, sending...
I20260927 23:04:19.531210 8142 ts_manager.cc:194] Registered new tserver with Master: 192d9a99420e49c3a9133e6cefbbead1 (127.3.240.195:43315)
I20260927 23:04:19.531893 8142 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:40073
I20260927 23:04:19.534226 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:04:19.548815 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.538140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.3.240.254:33259)
0927 23:04:19.538257 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.538261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.538672 (+ 411us) server_negotiation.cc:407] Connection header received
0927 23:04:19.538825 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.538835 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.538881 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.538981 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:19.540188 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.540757 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.541880 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.542282 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.542406 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.544472 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:19.544476 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:19.544479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:19.544502 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:19.546567 (+ 2065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.547136 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.547141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.547144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.547253 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.547601 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.547605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.547608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.547854 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:19.548100 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.548377 (+ 277us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.548547 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:04:19.551934 8142 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:19.558141 8142 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:19.582840 8607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.568532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:36755 (local address 127.0.0.1:54594)
0927 23:04:19.569155 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.569202 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:19.569336 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:19.570211 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:19.570218 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:19.570250 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:19.570664 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.570681 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.574027 (+ 3346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.574034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.578213 (+ 4179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.578231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.581889 (+ 3658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.581894 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.582061 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.582103 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 23:04:19.582225 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 23:04:19.582422 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":251,"threads_started":1}
I20260927 23:04:19.582940 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.568787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.3.240.194:36755)
0927 23:04:19.569310 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.569318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.569346 (+ 28us) server_negotiation.cc:407] Connection header received
0927 23:04:19.569603 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.569613 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.569867 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.570183 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:19.570885 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.572688 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.578425 (+ 5737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.580358 (+ 1933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.580486 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.580739 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.582296 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.582513 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":1035,"threads_started":1}
I20260927 23:04:19.586479 8398 tablet_service.cc:1467] Processing CreateTablet for tablet cef5ecd2c0f44ba99491deac39626b3c (DEFAULT_TABLE table=test-table [id=eaf0414b032a4a1d80d8d76c7871f596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:19.587854 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef5ecd2c0f44ba99491deac39626b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:19.589011 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.569512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43315 (local address 127.0.0.1:59066)
0927 23:04:19.570845 (+ 1333us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.570867 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:19.570961 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:19.582570 (+ 11609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:19.582575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:19.582597 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:19.582984 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.583024 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.584750 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.584757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.586615 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.586631 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.588243 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.588249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.588397 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.588426 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 23:04:19.588531 (+ 105us) client_negotiation.cc:240] Negotiation successful
0927 23:04:19.588648 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1206,"thread_start_us":1041,"threads_started":1}
I20260927 23:04:19.589092 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.569874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.3.240.195:43315)
0927 23:04:19.581834 (+ 11960us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.581841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.581874 (+ 33us) server_negotiation.cc:407] Connection header received
0927 23:04:19.581955 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.581973 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.582179 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.582336 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:19.583202 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.584633 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.586751 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.588068 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.588187 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.588380 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.588583 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.588740 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11811,"thread_start_us":11680,"threads_started":1}
I20260927 23:04:19.590395 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.570917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.0.0.1:32954)
0927 23:04:19.581002 (+ 10085us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.581030 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:19.581165 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:19.584354 (+ 3189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:19.584358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:19.584382 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:19.584688 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.584698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.586432 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.586437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.587848 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.587859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.589653 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.589662 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.589813 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.589841 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 23:04:19.589939 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 23:04:19.590069 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9897,"thread_start_us":10331,"threads_started":1}
I20260927 23:04:19.590516 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.571364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.3.240.193:39103)
0927 23:04:19.583890 (+ 12526us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.583897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.583922 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:04:19.583999 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.584006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.584185 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.584338 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:19.584867 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.586296 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.588058 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.589454 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.589589 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.589747 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.589999 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.590267 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12373,"thread_start_us":12294,"threads_started":1}
I20260927 23:04:19.592828 8530 tablet_service.cc:1467] Processing CreateTablet for tablet cef5ecd2c0f44ba99491deac39626b3c (DEFAULT_TABLE table=test-table [id=eaf0414b032a4a1d80d8d76c7871f596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:19.594182 8530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef5ecd2c0f44ba99491deac39626b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:19.594873 8265 tablet_service.cc:1467] Processing CreateTablet for tablet cef5ecd2c0f44ba99491deac39626b3c (DEFAULT_TABLE table=test-table [id=eaf0414b032a4a1d80d8d76c7871f596]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:19.596119 8265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef5ecd2c0f44ba99491deac39626b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:19.604787 8613 tablet_bootstrap.cc:492] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: Bootstrap starting.
I20260927 23:04:19.608171 8613 tablet_bootstrap.cc:654] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:19.609292 8613 log.cc:824] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:19.609450 8615 tablet_bootstrap.cc:492] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70: Bootstrap starting.
I20260927 23:04:19.613408 8614 tablet_bootstrap.cc:492] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: Bootstrap starting.
I20260927 23:04:19.613529 8615 tablet_bootstrap.cc:654] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:19.615067 8615 log.cc:824] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:19.616428 8614 tablet_bootstrap.cc:654] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:19.617347 8613 tablet_bootstrap.cc:492] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: No bootstrap required, opened a new log
I20260927 23:04:19.617592 8613 ts_tablet_manager.cc:1397] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260927 23:04:19.617588 8614 log.cc:824] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:19.619557 8615 tablet_bootstrap.cc:492] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70: No bootstrap required, opened a new log
I20260927 23:04:19.619794 8615 ts_tablet_manager.cc:1397] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260927 23:04:19.619916 8614 tablet_bootstrap.cc:492] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: No bootstrap required, opened a new log
I20260927 23:04:19.620154 8614 ts_tablet_manager.cc:1397] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260927 23:04:19.622308 8330 heartbeater.cc:503] Master 127.3.240.254:33259 was elected leader, sending a full tablet report...
I20260927 23:04:19.627162 8614 raft_consensus.cc:348] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.627230 8613 raft_consensus.cc:348] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.627480 8614 raft_consensus.cc:374] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:19.627558 8614 raft_consensus.cc:729] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 192d9a99420e49c3a9133e6cefbbead1, State: Initialized, Role: FOLLOWER
I20260927 23:04:19.627630 8613 raft_consensus.cc:374] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:19.627739 8613 raft_consensus.cc:729] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4e956de80e643f391c75d0330b9d9fa, State: Initialized, Role: FOLLOWER
I20260927 23:04:19.628149 8614 consensus_queue.cc:260] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.628309 8613 consensus_queue.cc:260] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.629148 8595 heartbeater.cc:503] Master 127.3.240.254:33259 was elected leader, sending a full tablet report...
I20260927 23:04:19.629607 8614 ts_tablet_manager.cc:1428] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: Time spent starting tablet: real 0.009s user 0.007s sys 0.003s
I20260927 23:04:19.633733 8615 raft_consensus.cc:348] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.634212 8615 raft_consensus.cc:374] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:19.634317 8615 raft_consensus.cc:729] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4609b5c8cf0f451e929b9e5195222f70, State: Initialized, Role: FOLLOWER
I20260927 23:04:19.634990 8615 consensus_queue.cc:260] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.637085 8615 ts_tablet_manager.cc:1428] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70: Time spent starting tablet: real 0.017s user 0.011s sys 0.002s
I20260927 23:04:19.638015 8463 heartbeater.cc:503] Master 127.3.240.254:33259 was elected leader, sending a full tablet report...
I20260927 23:04:19.638518 8613 ts_tablet_manager.cc:1428] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: Time spent starting tablet: real 0.021s user 0.015s sys 0.004s
I20260927 23:04:19.652576 8620 raft_consensus.cc:482] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:19.652848 8620 raft_consensus.cc:504] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.654399 8620 leader_election.cc:288] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 192d9a99420e49c3a9133e6cefbbead1 (127.3.240.195:43315), 4609b5c8cf0f451e929b9e5195222f70 (127.3.240.193:39103)
I20260927 23:04:19.660723 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.654739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:58633 (local address 127.3.240.195:43315)
0927 23:04:19.654943 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.654947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.655572 (+ 625us) server_negotiation.cc:407] Connection header received
0927 23:04:19.655897 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.655902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.655976 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.656039 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:19.656662 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.657336 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.659092 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.659847 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.660029 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.660103 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.660361 (+ 258us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.660479 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:04:19.660885 8622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.654739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43315 (local address 127.3.240.194:58633)
0927 23:04:19.655477 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.655496 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:19.655687 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:19.656187 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:19.656190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:19.656209 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:19.656528 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.656536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.657454 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.657462 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.658909 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.658923 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.660059 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.660065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.660209 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.660230 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 23:04:19.660330 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 23:04:19.660446 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":1,"spinlock_wait_cycles":17280,"thread_start_us":393,"threads_started":1}
I20260927 23:04:19.661556 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.655807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42619 (local address 127.3.240.193:39103)
0927 23:04:19.656026 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.656032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.656250 (+ 218us) server_negotiation.cc:407] Connection header received
0927 23:04:19.656484 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.656489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.656557 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.656649 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:19.657194 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.657922 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.659746 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.660543 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.660740 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.660819 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.661246 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.661356 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 23:04:19.661649 8623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.655534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.3.240.194:42619)
0927 23:04:19.656150 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.656164 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:19.656341 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:19.656674 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:19.656678 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:19.656702 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:19.657034 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.657043 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.658060 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.658066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.659588 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:19.659600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.660769 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.660776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:19.661032 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.661088 (+ 56us) client_negotiation.cc:769] Sending connection context
0927 23:04:19.661195 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 23:04:19.661287 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":1,"thread_start_us":280,"threads_started":1}
I20260927 23:04:19.661964 8550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef5ecd2c0f44ba99491deac39626b3c" candidate_uuid: "e4e956de80e643f391c75d0330b9d9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "192d9a99420e49c3a9133e6cefbbead1" is_pre_election: true
I20260927 23:04:19.662616 8550 raft_consensus.cc:2454] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e956de80e643f391c75d0330b9d9fa in term 0.
I20260927 23:04:19.662722 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef5ecd2c0f44ba99491deac39626b3c" candidate_uuid: "e4e956de80e643f391c75d0330b9d9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4609b5c8cf0f451e929b9e5195222f70" is_pre_election: true
I20260927 23:04:19.663298 8285 raft_consensus.cc:2454] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4e956de80e643f391c75d0330b9d9fa in term 0.
I20260927 23:04:19.663348 8352 leader_election.cc:302] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192d9a99420e49c3a9133e6cefbbead1, e4e956de80e643f391c75d0330b9d9fa; no voters:
I20260927 23:04:19.663721 8620 raft_consensus.cc:2781] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:19.663882 8620 raft_consensus.cc:482] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:19.663996 8620 raft_consensus.cc:3037] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:19.666947 8620 raft_consensus.cc:504] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.667680 8620 leader_election.cc:288] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [CANDIDATE]: Term 1 election: Requested vote from peers 192d9a99420e49c3a9133e6cefbbead1 (127.3.240.195:43315), 4609b5c8cf0f451e929b9e5195222f70 (127.3.240.193:39103)
I20260927 23:04:19.668344 8550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef5ecd2c0f44ba99491deac39626b3c" candidate_uuid: "e4e956de80e643f391c75d0330b9d9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "192d9a99420e49c3a9133e6cefbbead1"
I20260927 23:04:19.668493 8550 raft_consensus.cc:3037] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:19.668431 8285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef5ecd2c0f44ba99491deac39626b3c" candidate_uuid: "e4e956de80e643f391c75d0330b9d9fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4609b5c8cf0f451e929b9e5195222f70"
I20260927 23:04:19.668632 8285 raft_consensus.cc:3037] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:19.670691 8550 raft_consensus.cc:2454] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e956de80e643f391c75d0330b9d9fa in term 1.
I20260927 23:04:19.670907 8285 raft_consensus.cc:2454] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4e956de80e643f391c75d0330b9d9fa in term 1.
I20260927 23:04:19.671056 8352 leader_election.cc:302] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 192d9a99420e49c3a9133e6cefbbead1, e4e956de80e643f391c75d0330b9d9fa; no voters:
I20260927 23:04:19.671319 8620 raft_consensus.cc:2781] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:19.671726 8620 raft_consensus.cc:686] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 1 LEADER]: Becoming Leader. State: Replica: e4e956de80e643f391c75d0330b9d9fa, State: Running, Role: LEADER
I20260927 23:04:19.672173 8620 consensus_queue.cc:237] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } }
I20260927 23:04:19.676208 8141 catalog_manager.cc:5495] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa reported cstate change: term changed from 0 to 1, leader changed from <none> to e4e956de80e643f391c75d0330b9d9fa (127.3.240.194). New cstate: current_term: 1 leader_uuid: "e4e956de80e643f391c75d0330b9d9fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4e956de80e643f391c75d0330b9d9fa" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 } health_report { overall_health: UNKNOWN } } }
I20260927 23:04:19.701094 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.696360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.3.240.194:36755)
0927 23:04:19.696569 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.696574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.696589 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:04:19.696712 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.696715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.696820 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.696947 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:19.697253 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.697944 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.699260 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.699673 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.699777 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.700190 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:19.700668 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:19.700770 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.700839 (+ 69us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.700897 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260927 23:04:19.709025 8285 raft_consensus.cc:1261] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 1 FOLLOWER]: Refusing update from remote peer e4e956de80e643f391c75d0330b9d9fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:19.709054 8550 raft_consensus.cc:1261] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 1 FOLLOWER]: Refusing update from remote peer e4e956de80e643f391c75d0330b9d9fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:19.710017 8624 consensus_queue.cc:1035] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "192d9a99420e49c3a9133e6cefbbead1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:19.710407 8620 consensus_queue.cc:1035] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4609b5c8cf0f451e929b9e5195222f70" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:19.735160 8141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:19.735458 8141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40068:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:19.739195 8141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9598d0ae122441c4bd3cb1ab6209d64c: Sending DeleteTablet for 3 replicas of tablet cef5ecd2c0f44ba99491deac39626b3c
I20260927 23:04:19.740219 8398 tablet_service.cc:1514] Processing DeleteTablet for tablet cef5ecd2c0f44ba99491deac39626b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:19 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:54594
I20260927 23:04:19.740576 8530 tablet_service.cc:1514] Processing DeleteTablet for tablet cef5ecd2c0f44ba99491deac39626b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:19 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:59066
I20260927 23:04:19.741163 8636 tablet_replica.cc:330] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: stopping tablet replica
I20260927 23:04:19.741278 8265 tablet_service.cc:1514] Processing DeleteTablet for tablet cef5ecd2c0f44ba99491deac39626b3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:19 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:32954
I20260927 23:04:19.741580 8636 raft_consensus.cc:2229] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:19.741815 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 8203
I20260927 23:04:19.742276 8636 raft_consensus.cc:2258] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:19.744899 8638 tablet_replica.cc:330] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70: stopping tablet replica
I20260927 23:04:19.745343 8638 raft_consensus.cc:2229] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:19.745814 8638 raft_consensus.cc:2258] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:19.748606 8636 ts_tablet_manager.cc:1905] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:19.750787 8638 ts_tablet_manager.cc:1905] T cef5ecd2c0f44ba99491deac39626b3c P 4609b5c8cf0f451e929b9e5195222f70: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 23:04:19 dist-test-slave-lmfp krb5kdc[8098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:04:19.762568 8637 tablet_replica.cc:330] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: stopping tablet replica
I20260927 23:04:19.763106 8637 raft_consensus.cc:2229] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:19.763645 8637 raft_consensus.cc:2258] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:19.765118 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.743250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.3.240.193:39103)
0927 23:04:19.743409 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.743415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.746523 (+ 3108us) server_negotiation.cc:407] Connection header received
0927 23:04:19.746775 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.746782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.746863 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.746957 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:19.748614 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.749754 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.751609 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.752329 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.752424 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.757563 (+ 5139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:19.757580 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:19.757597 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:19.757634 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:19.761538 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.762396 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.762405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.762408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.762551 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.763100 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.763105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.763109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.763539 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:19.763808 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.764493 (+ 685us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.764700 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260927 23:04:19.766711 8637 ts_tablet_manager.cc:1905] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:19.766779 8636 ts_tablet_manager.cc:1918] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:19.767024 8636 log.cc:1196] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/cef5ecd2c0f44ba99491deac39626b3c
I20260927 23:04:19.767618 8636 ts_tablet_manager.cc:1939] T cef5ecd2c0f44ba99491deac39626b3c P e4e956de80e643f391c75d0330b9d9fa: Deleting consensus metadata
I20260927 23:04:19.768075 8325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:19.769706 8127 catalog_manager.cc:4841] TS e4e956de80e643f391c75d0330b9d9fa (127.3.240.194:36755): tablet cef5ecd2c0f44ba99491deac39626b3c (table test-table [id=eaf0414b032a4a1d80d8d76c7871f596]) successfully deleted
I20260927 23:04:19.774044 8637 ts_tablet_manager.cc:1918] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:19.774261 8637 log.cc:1196] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/cef5ecd2c0f44ba99491deac39626b3c
I20260927 23:04:19.774820 8637 ts_tablet_manager.cc:1939] T cef5ecd2c0f44ba99491deac39626b3c P 192d9a99420e49c3a9133e6cefbbead1: Deleting consensus metadata
I20260927 23:04:19.776494 8127 catalog_manager.cc:4841] TS 192d9a99420e49c3a9133e6cefbbead1 (127.3.240.195:43315): tablet cef5ecd2c0f44ba99491deac39626b3c (table test-table [id=eaf0414b032a4a1d80d8d76c7871f596]) successfully deleted
I20260927 23:04:19.875774 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 8203
W20260927 23:04:19.892755 8127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:39103 (error 108)
I20260927 23:04:19.892964 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 8336
I20260927 23:04:19.895794 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.895353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.0.0.1:32964)
0927 23:04:19.895515 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.895626 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 23:04:19.896145 8127 catalog_manager.cc:4584] TS 4609b5c8cf0f451e929b9e5195222f70 (127.3.240.193:39103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cef5ecd2c0f44ba99491deac39626b3c: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39103: connect: Connection refused (error 111)
Sep 27 23:04:19 dist-test-slave-lmfp krb5kdc[8098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:04:19.909312 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.894299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.3.240.194:36755)
0927 23:04:19.894548 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:19.894554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:19.894570 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:04:19.894632 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:19.894639 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:19.894724 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:19.894816 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:19.896298 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.897512 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.898792 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:19.899443 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:19.899560 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:19.903596 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:19.903612 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:19.903626 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:19.903663 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:19.906608 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.907379 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.907386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.907388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.907533 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:19.907910 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:19.907919 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:19.907920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:19.908251 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:19.908576 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:19.908778 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 23:04:19.908981 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:04:19.909945 8458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:19.949707 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.949201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.0.0.1:32970)
0927 23:04:19.949418 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.949538 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 23:04:19.950649 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:19.950286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.0.0.1:32980)
0927 23:04:19.950431 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:19.950512 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260927 23:04:20.011428 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:20.010826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.0.0.1:32994)
0927 23:04:20.011179 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:20.011269 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260927 23:04:20.012405 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:20.012087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.0.0.1:33000)
0927 23:04:20.012235 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:20.012317 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 23:04:20.017908 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 8336
I20260927 23:04:20.035486 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 8468
Sep 27 23:04:20 dist-test-slave-lmfp krb5kdc[8098](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:04:20.049937 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:20.036231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.3.240.195:43315)
0927 23:04:20.036400 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:20.036407 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:20.036427 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:04:20.036556 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:20.036562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:20.036636 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:20.036756 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:20.037951 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:20.038984 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:20.040104 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:20.040656 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:20.040758 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:20.044740 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:20.044756 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:20.044775 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:20.044814 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:20.047830 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:20.048436 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:20.048443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:20.048445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:20.048571 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:20.048841 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:20.048844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:20.048845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:20.049159 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:20.049348 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:20.049575 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 23:04:20.049709 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 23:04:20.050596 8590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:20.093411 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:20.092984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.0.0.1:33002)
0927 23:04:20.093186 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:20.093271 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 23:04:20.094414 8611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:20.094068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39103 (local address 127.0.0.1:33012)
0927 23:04:20.094255 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:20.094317 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 23:04:20.133380 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 8468
I20260927 23:04:20.149828 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 8111
I20260927 23:04:20.150944 8172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:20.254940 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 8111
2026-09-27T23:04:20Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3872 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-27T23:04:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:20Z Disabled control of system clock
I20260927 23:04:20.317658 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:34583
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:34583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 23:04:20.620232 8648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:20.620492 8648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:20.620644 8648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:20.627589 8648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:20.627681 8648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:20.627761 8648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:20.627861 8648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:20.627944 8648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:20.636786 8648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:34583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:34583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:20.638028 8648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:20.639767 8648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:20.651293 8654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:20.651413 8656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:20.651889 8653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:20.652000 8648 server_base.cc:1034] running on GCE node
I20260927 23:04:20.652838 8648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:20.654175 8648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:20.655356 8648 hybrid_clock.cc:648] HybridClock initialized: now 1790550260655308 us; error 41 us; skew 500 ppm
I20260927 23:04:20.655690 8648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:20.657835 8648 webserver.cc:460] Webserver started at http://127.3.240.254:38073/ using document root <none> and password file <none>
I20260927 23:04:20.658496 8648 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:20.658602 8648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:20.658912 8648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:20.661566 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "8f9ec92d02c44514b8fef81af15f6f09"
format_stamp: "Formatted at 2026-09-27 23:04:20 on dist-test-slave-lmfp"
server_key: "08d4f4778d4cc3b327c3660301132ca4"
server_key_iv: "35ae95f8920269d2a7cb05df12fe2f2a"
server_key_version: "encryptionkey@0"
I20260927 23:04:20.662288 8648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "8f9ec92d02c44514b8fef81af15f6f09"
format_stamp: "Formatted at 2026-09-27 23:04:20 on dist-test-slave-lmfp"
server_key: "08d4f4778d4cc3b327c3660301132ca4"
server_key_iv: "35ae95f8920269d2a7cb05df12fe2f2a"
server_key_version: "encryptionkey@0"
I20260927 23:04:20.667112 8648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260927 23:04:20.670943 8662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:20.672559 8648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:04:20.672717 8648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "8f9ec92d02c44514b8fef81af15f6f09"
format_stamp: "Formatted at 2026-09-27 23:04:20 on dist-test-slave-lmfp"
server_key: "08d4f4778d4cc3b327c3660301132ca4"
server_key_iv: "35ae95f8920269d2a7cb05df12fe2f2a"
server_key_version: "encryptionkey@0"
I20260927 23:04:20.672917 8648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:20.678071 8648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:20.678759 8648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:20.679152 8648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:20.695403 8648 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:34583
I20260927 23:04:20.695421 8713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:34583 every 8 connection(s)
I20260927 23:04:20.696669 8648 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:20.699980 8714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:20.702762 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 8648
I20260927 23:04:20.702950 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:20.703729 4035 external_mini_cluster.cc:1428] Setting key 22fede5da766e9990de94c292b39068e
I20260927 23:04:20.711472 8714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09: Bootstrap starting.
I20260927 23:04:20.711489 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:20.705725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.3.240.254:34583)
0927 23:04:20.706663 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:20.706673 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:20.706711 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:04:20.708268 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:20.708288 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:20.708689 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:20.709053 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:20.710011 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:20.710016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:20.710027 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:20.710039 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:20.710089 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:20.710103 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:20.710304 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:20.710430 (+ 126us) server_negotiation.cc:299] Negotiation successful
0927 23:04:20.710647 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":348,"threads_started":1}
I20260927 23:04:20.714917 8714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:20.715868 8714 log.cc:824] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:20.718238 8714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09: No bootstrap required, opened a new log
I20260927 23:04:20.725440 8714 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ec92d02c44514b8fef81af15f6f09" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34583 } }
I20260927 23:04:20.725726 8714 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:20.725807 8714 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9ec92d02c44514b8fef81af15f6f09, State: Initialized, Role: FOLLOWER
I20260927 23:04:20.726358 8714 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ec92d02c44514b8fef81af15f6f09" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34583 } }
I20260927 23:04:20.726593 8714 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:20.726711 8714 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:20.726869 8714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:20.729039 8714 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ec92d02c44514b8fef81af15f6f09" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34583 } }
I20260927 23:04:20.729498 8714 leader_election.cc:302] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f9ec92d02c44514b8fef81af15f6f09; no voters:
I20260927 23:04:20.730103 8714 leader_election.cc:288] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:20.730262 8719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:20.730973 8719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [term 1 LEADER]: Becoming Leader. State: Replica: 8f9ec92d02c44514b8fef81af15f6f09, State: Running, Role: LEADER
I20260927 23:04:20.731464 8719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ec92d02c44514b8fef81af15f6f09" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34583 } }
I20260927 23:04:20.732122 8714 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:20.733906 8720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f9ec92d02c44514b8fef81af15f6f09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ec92d02c44514b8fef81af15f6f09" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34583 } } }
I20260927 23:04:20.733999 8721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f9ec92d02c44514b8fef81af15f6f09. Latest consensus state: current_term: 1 leader_uuid: "8f9ec92d02c44514b8fef81af15f6f09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9ec92d02c44514b8fef81af15f6f09" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34583 } } }
I20260927 23:04:20.734273 8720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:20.734269 8721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09 [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:20.735752 8726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:20.740401 8726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:20.747805 8726 catalog_manager.cc:1347] Generated new cluster ID: 759495ae2e194f33a9197de538bba391
I20260927 23:04:20.747910 8726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:20.762070 8726 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:20.762488 8726 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:20.773572 8726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09: Generated new TSK 0
I20260927 23:04:20.774631 8726 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:04:20.782850 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34583
--builtin_ntp_servers=127.3.240.212:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260927 23:04:21.088138 8738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:21.088389 8738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:21.088488 8738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:21.095248 8738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:21.095333 8738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:21.095432 8738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:21.103948 8738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:21.105067 8738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:21.106762 8738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:21.120994 8743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:21.120929 8746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:21.120955 8744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:21.121922 8738 server_base.cc:1034] running on GCE node
I20260927 23:04:21.122504 8738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:21.123538 8738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:21.124791 8738 hybrid_clock.cc:648] HybridClock initialized: now 1790550261124757 us; error 57 us; skew 500 ppm
I20260927 23:04:21.125129 8738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:21.127282 8738 webserver.cc:460] Webserver started at http://127.3.240.193:34119/ using document root <none> and password file <none>
I20260927 23:04:21.127930 8738 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:21.128060 8738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:21.128357 8738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:21.131172 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "e7c21015921b447e86c432cde4b727eb"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "ea2f3fe8fa61a787fdeaa6c6a4700b3a"
server_key_iv: "8b08f1f619a6e8d120988cd634de57ad"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.132031 8738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "e7c21015921b447e86c432cde4b727eb"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "ea2f3fe8fa61a787fdeaa6c6a4700b3a"
server_key_iv: "8b08f1f619a6e8d120988cd634de57ad"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.136986 8738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 23:04:21.140450 8752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.141759 8738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 23:04:21.141939 8738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "e7c21015921b447e86c432cde4b727eb"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "ea2f3fe8fa61a787fdeaa6c6a4700b3a"
server_key_iv: "8b08f1f619a6e8d120988cd634de57ad"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.142169 8738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:21.148200 8738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:21.148936 8738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:21.149204 8738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:21.150094 8738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:21.151494 8738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:21.151611 8738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.151752 8738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:21.151836 8738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.184360 8738 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:35391
I20260927 23:04:21.184410 8864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:35391 every 8 connection(s)
I20260927 23:04:21.185567 8738 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:21.189329 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 8738
I20260927 23:04:21.189509 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:21.190086 4035 external_mini_cluster.cc:1428] Setting key c00515c2d04b8dadd7c08cec8e5a2110
I20260927 23:04:21.192996 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34583
--builtin_ntp_servers=127.3.240.212:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 23:04:21.193761 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:21.188008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:40457 (local address 127.3.240.254:34583)
0927 23:04:21.188161 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:21.188165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:21.189458 (+ 1293us) server_negotiation.cc:407] Connection header received
0927 23:04:21.191417 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:21.191422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:21.191483 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:21.191578 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:21.193004 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:21.193008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:21.193010 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:21.193024 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:21.193053 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:21.193064 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:21.193217 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:21.193452 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 23:04:21.193529 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 23:04:21.194650 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:21.188440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34583 (local address 127.3.240.193:40457)
0927 23:04:21.189288 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:21.189341 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:21.191169 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:21.191829 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:21.191837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:21.191968 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:21.192624 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:04:21.192629 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:21.192698 (+ 69us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:04:21.192807 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:21.193256 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:21.193270 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:21.193276 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 23:04:21.193524 (+ 248us) client_negotiation.cc:240] Negotiation successful
0927 23:04:21.193745 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":241,"threads_started":1}
I20260927 23:04:21.196113 8865 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34583
I20260927 23:04:21.196367 8865 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:21.196915 8865 heartbeater.cc:511] Master 127.3.240.254:34583 requested a full tablet report, sending...
I20260927 23:04:21.199168 8679 ts_manager.cc:194] Registered new tserver with Master: e7c21015921b447e86c432cde4b727eb (127.3.240.193:35391)
W20260927 23:04:21.504596 8868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:21.504884 8868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:21.504976 8868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:21.512053 8868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:21.512177 8868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:21.512290 8868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:21.520927 8868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:21.522121 8868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:21.523764 8868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:21.536393 8874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:21.536448 8876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:21.536489 8873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:21.537333 8868 server_base.cc:1034] running on GCE node
I20260927 23:04:21.537882 8868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:21.538762 8868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:21.539945 8868 hybrid_clock.cc:648] HybridClock initialized: now 1790550261539880 us; error 59 us; skew 500 ppm
I20260927 23:04:21.540206 8868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:21.542284 8868 webserver.cc:460] Webserver started at http://127.3.240.194:33277/ using document root <none> and password file <none>
I20260927 23:04:21.542896 8868 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:21.543030 8868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:21.543327 8868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:21.545997 8868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "82f4a4dd85014052854e4e267b8be9ab"
server_key_iv: "e2d2dd3cf268e73981bf9037b4973a60"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.546761 8868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "82f4a4dd85014052854e4e267b8be9ab"
server_key_iv: "e2d2dd3cf268e73981bf9037b4973a60"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.551282 8868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 23:04:21.554270 8882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.555518 8868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:21.555656 8868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "82f4a4dd85014052854e4e267b8be9ab"
server_key_iv: "e2d2dd3cf268e73981bf9037b4973a60"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.555857 8868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:21.561084 8868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:21.561838 8868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:21.562111 8868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:21.562958 8868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:21.564213 8868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:21.564314 8868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.564437 8868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:21.564523 8868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.595224 8868 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:39339
I20260927 23:04:21.595270 8994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:39339 every 8 connection(s)
I20260927 23:04:21.596547 8868 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:21.600682 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 8868
I20260927 23:04:21.600821 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:21.601238 4035 external_mini_cluster.cc:1428] Setting key a8de8ef7af2b6a78af64640c51a1c381
I20260927 23:04:21.604110 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34583
--builtin_ntp_servers=127.3.240.212:39163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260927 23:04:21.604718 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:21.598922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:50507 (local address 127.3.240.254:34583)
0927 23:04:21.599104 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:21.599108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:21.600497 (+ 1389us) server_negotiation.cc:407] Connection header received
0927 23:04:21.602340 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:21.602344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:21.602386 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:21.602482 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:21.603944 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:21.603949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:21.603954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:21.603970 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:21.604027 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:21.604038 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:21.604164 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:21.604396 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 23:04:21.604467 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 23:04:21.605561 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:21.599494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34583 (local address 127.3.240.194:50507)
0927 23:04:21.600333 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:21.600364 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:21.602064 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:21.602766 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:21.602787 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:21.602927 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:21.603663 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:04:21.603665 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:21.603698 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:04:21.603772 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:21.604140 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:21.604149 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:21.604155 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 23:04:21.604463 (+ 308us) client_negotiation.cc:240] Negotiation successful
0927 23:04:21.604684 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":255,"threads_started":1}
I20260927 23:04:21.606966 8995 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34583
I20260927 23:04:21.607288 8995 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:21.607792 8995 heartbeater.cc:511] Master 127.3.240.254:34583 requested a full tablet report, sending...
I20260927 23:04:21.609050 8679 ts_manager.cc:194] Registered new tserver with Master: b17e185b1c4c4c748f1a5d8c1cba6fc8 (127.3.240.194:39339)
W20260927 23:04:21.905700 8998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:21.906006 8998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:21.906165 8998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:21.912665 8998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:21.912782 8998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:21.912978 8998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:21.921545 8998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:21.922784 8998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:21.924528 8998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:21.937007 9003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:21.937086 9004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:21.937537 9006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:21.937903 8998 server_base.cc:1034] running on GCE node
I20260927 23:04:21.938494 8998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:21.939374 8998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:21.940555 8998 hybrid_clock.cc:648] HybridClock initialized: now 1790550261940488 us; error 59 us; skew 500 ppm
I20260927 23:04:21.940796 8998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:21.942772 8998 webserver.cc:460] Webserver started at http://127.3.240.195:46487/ using document root <none> and password file <none>
I20260927 23:04:21.943413 8998 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:21.943492 8998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:21.943730 8998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:21.946295 8998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "46ccbdaeada3465ba8cad384b3d81ea2"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "1c35ca1033bd2c21d23189cd8eaa5547"
server_key_iv: "c59cedf8f693cbd9e6ab7790b86fa7ee"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.947136 8998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "46ccbdaeada3465ba8cad384b3d81ea2"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "1c35ca1033bd2c21d23189cd8eaa5547"
server_key_iv: "c59cedf8f693cbd9e6ab7790b86fa7ee"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.951766 8998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 23:04:21.954926 9012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.956178 8998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 23:04:21.956296 8998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "46ccbdaeada3465ba8cad384b3d81ea2"
format_stamp: "Formatted at 2026-09-27 23:04:21 on dist-test-slave-lmfp"
server_key: "1c35ca1033bd2c21d23189cd8eaa5547"
server_key_iv: "c59cedf8f693cbd9e6ab7790b86fa7ee"
server_key_version: "encryptionkey@0"
I20260927 23:04:21.956488 8998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:21.962177 8998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:21.962863 8998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:21.963142 8998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:21.963904 8998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:21.965114 8998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:21.965194 8998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.965260 8998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:21.965304 8998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:21.996672 8998 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:46481
I20260927 23:04:21.996742 9124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:46481 every 8 connection(s)
I20260927 23:04:21.997949 8998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:22.001479 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 8998
I20260927 23:04:22.001660 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:22.002307 4035 external_mini_cluster.cc:1428] Setting key 361fe03a1997060bf81ba3e7a4807f6d
I20260927 23:04:22.006887 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.000272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:53101 (local address 127.3.240.254:34583)
0927 23:04:22.000451 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.000455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.002834 (+ 2379us) server_negotiation.cc:407] Connection header received
0927 23:04:22.004906 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.004909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.004955 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.005069 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.006323 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.006328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.006331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.006345 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.006374 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.006384 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.006498 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.006665 (+ 167us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.006709 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:04:22.007781 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.000695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34583 (local address 127.3.240.195:53101)
0927 23:04:22.002665 (+ 1970us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.002702 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:22.004612 (+ 1910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:22.005320 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:22.005333 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:22.005458 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:22.006070 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:04:22.006072 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:22.006106 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:04:22.006180 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:22.006464 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:22.006479 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:22.006485 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 23:04:22.006709 (+ 224us) client_negotiation.cc:240] Negotiation successful
0927 23:04:22.006881 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1524,"thread_start_us":1370,"threads_started":1}
I20260927 23:04:22.009035 9125 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34583
I20260927 23:04:22.009307 9125 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:22.009793 9125 heartbeater.cc:511] Master 127.3.240.254:34583 requested a full tablet report, sending...
I20260927 23:04:22.011059 8679 ts_manager.cc:194] Registered new tserver with Master: 46ccbdaeada3465ba8cad384b3d81ea2 (127.3.240.195:46481)
I20260927 23:04:22.017248 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:04:22.023696 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.021418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.3.240.254:34583)
0927 23:04:22.021568 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.021572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.022110 (+ 538us) server_negotiation.cc:407] Connection header received
0927 23:04:22.022215 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.022218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.022258 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.022335 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.023254 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.023258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.023260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.023272 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.023296 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.023306 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.023393 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.023476 (+ 83us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.023515 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:04:22.027202 8679 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:22.034039 8679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:22.053627 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.045973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56082 (local address 127.3.240.193:35391)
0927 23:04:22.050501 (+ 4528us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.050508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.050532 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:04:22.050588 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.050594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.050753 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.051899 (+ 1146us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.052706 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.052725 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.052737 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.052760 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.052812 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.052832 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.053115 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.053165 (+ 50us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.053295 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4366,"thread_start_us":4802,"threads_started":1}
I20260927 23:04:22.053882 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.046449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:46481 (local address 127.0.0.1:42344)
0927 23:04:22.047485 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.047501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:22.047598 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:22.050821 (+ 3223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:22.050830 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:22.050926 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:22.052042 (+ 1116us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:04:22.052062 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:22.052080 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:04:22.052122 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:22.052878 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:22.052887 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:22.052894 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 23:04:22.053339 (+ 445us) client_negotiation.cc:240] Negotiation successful
0927 23:04:22.053421 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":928,"threads_started":1}
I20260927 23:04:22.054097 9137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.045847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35391 (local address 127.0.0.1:56082)
0927 23:04:22.046588 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.046693 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:22.046821 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:22.050897 (+ 4076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:22.050901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:22.050925 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:22.051698 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:04:22.051703 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:22.051746 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:04:22.051835 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:22.052964 (+ 1129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:22.052968 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:22.052974 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 23:04:22.053652 (+ 678us) client_negotiation.cc:240] Negotiation successful
0927 23:04:22.053700 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":291,"threads_started":1}
I20260927 23:04:22.054988 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.047100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.3.240.195:46481)
0927 23:04:22.050262 (+ 3162us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.050272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.050308 (+ 36us) server_negotiation.cc:407] Connection header received
0927 23:04:22.050388 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.050395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.050624 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.051058 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.052638 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.052648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.052662 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.052678 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.052740 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.052758 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.054440 (+ 1682us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.054524 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.054612 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2998,"thread_start_us":2819,"threads_started":1}
I20260927 23:04:22.055338 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.047623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.3.240.194:39339)
0927 23:04:22.051504 (+ 3881us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.051515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.052138 (+ 623us) server_negotiation.cc:407] Connection header received
0927 23:04:22.052335 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.052343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.052540 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.052703 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.054344 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.054351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.054363 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.054380 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.054460 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.054484 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.054651 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.054869 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.054985 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3691,"thread_start_us":3541,"threads_started":1}
I20260927 23:04:22.055343 9140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.046973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:39339 (local address 127.0.0.1:44876)
0927 23:04:22.051916 (+ 4943us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.051932 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:22.052184 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:22.052729 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:22.052733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:22.052754 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:22.054138 (+ 1384us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:04:22.054143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:22.054157 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:04:22.054195 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:22.054706 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:22.054712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:22.054718 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 23:04:22.054845 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 23:04:22.054891 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4840,"thread_start_us":4673,"threads_started":1}
I20260927 23:04:22.058262 8800 tablet_service.cc:1467] Processing CreateTablet for tablet c863665461ed4466bb8b722998526552 (DEFAULT_TABLE table=test-table [id=a689a350b95c4335a62156f2ad961c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:22.058732 9060 tablet_service.cc:1467] Processing CreateTablet for tablet c863665461ed4466bb8b722998526552 (DEFAULT_TABLE table=test-table [id=a689a350b95c4335a62156f2ad961c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:22.059849 8800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c863665461ed4466bb8b722998526552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:22.059644 8930 tablet_service.cc:1467] Processing CreateTablet for tablet c863665461ed4466bb8b722998526552 (DEFAULT_TABLE table=test-table [id=a689a350b95c4335a62156f2ad961c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:22.060184 9060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c863665461ed4466bb8b722998526552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:22.060920 8930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c863665461ed4466bb8b722998526552. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:22.070819 9143 tablet_bootstrap.cc:492] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Bootstrap starting.
I20260927 23:04:22.073725 9144 tablet_bootstrap.cc:492] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: Bootstrap starting.
I20260927 23:04:22.073890 9143 tablet_bootstrap.cc:654] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:22.075201 9145 tablet_bootstrap.cc:492] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Bootstrap starting.
I20260927 23:04:22.076056 9143 log.cc:824] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:22.077293 9144 tablet_bootstrap.cc:654] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:22.078090 9145 tablet_bootstrap.cc:654] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:22.078408 9144 log.cc:824] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:22.079152 9143 tablet_bootstrap.cc:492] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: No bootstrap required, opened a new log
I20260927 23:04:22.079104 9145 log.cc:824] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:22.079406 9143 ts_tablet_manager.cc:1397] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 23:04:22.081199 9144 tablet_bootstrap.cc:492] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: No bootstrap required, opened a new log
I20260927 23:04:22.081413 9145 tablet_bootstrap.cc:492] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: No bootstrap required, opened a new log
I20260927 23:04:22.081453 9144 ts_tablet_manager.cc:1397] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 23:04:22.081640 9145 ts_tablet_manager.cc:1397] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:04:22.088274 9145 raft_consensus.cc:348] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.088603 9145 raft_consensus.cc:374] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:22.088716 9145 raft_consensus.cc:729] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17e185b1c4c4c748f1a5d8c1cba6fc8, State: Initialized, Role: FOLLOWER
I20260927 23:04:22.089289 9145 consensus_queue.cc:260] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.089785 9143 raft_consensus.cc:348] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.090340 8995 heartbeater.cc:503] Master 127.3.240.254:34583 was elected leader, sending a full tablet report...
I20260927 23:04:22.090780 9145 ts_tablet_manager.cc:1428] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260927 23:04:22.091005 9143 raft_consensus.cc:374] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:22.091194 9143 raft_consensus.cc:729] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46ccbdaeada3465ba8cad384b3d81ea2, State: Initialized, Role: FOLLOWER
I20260927 23:04:22.091265 9144 raft_consensus.cc:348] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.091671 9144 raft_consensus.cc:374] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:22.091773 9144 raft_consensus.cc:729] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c21015921b447e86c432cde4b727eb, State: Initialized, Role: FOLLOWER
I20260927 23:04:22.091776 9143 consensus_queue.cc:260] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.092285 9144 consensus_queue.cc:260] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.093416 8865 heartbeater.cc:503] Master 127.3.240.254:34583 was elected leader, sending a full tablet report...
I20260927 23:04:22.093894 9144 ts_tablet_manager.cc:1428] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 23:04:22.095192 9143 ts_tablet_manager.cc:1428] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260927 23:04:22.095674 9125 heartbeater.cc:503] Master 127.3.240.254:34583 was elected leader, sending a full tablet report...
W20260927 23:04:22.098269 8996 tablet.cc:2369] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:04:22.187668 8866 tablet.cc:2369] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:04:22.249605 9126 tablet.cc:2369] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:04:22.314361 9151 raft_consensus.cc:482] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:22.314759 9151 raft_consensus.cc:504] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.316459 9151 leader_election.cc:288] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b17e185b1c4c4c748f1a5d8c1cba6fc8 (127.3.240.194:39339), 46ccbdaeada3465ba8cad384b3d81ea2 (127.3.240.195:46481)
I20260927 23:04:22.319818 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.316953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:36241 (local address 127.3.240.194:39339)
0927 23:04:22.317263 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.317269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.317671 (+ 402us) server_negotiation.cc:407] Connection header received
0927 23:04:22.317890 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.317894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.317940 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.318047 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.319206 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.319212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.319216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.319235 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.319275 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.319286 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.319414 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.319495 (+ 81us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.319559 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260927 23:04:22.319903 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.316903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:36883 (local address 127.3.240.195:46481)
0927 23:04:22.317098 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.317106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.317922 (+ 816us) server_negotiation.cc:407] Connection header received
0927 23:04:22.318102 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.318107 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.318151 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.318258 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.319287 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.319298 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.319301 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.319315 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.319344 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.319354 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.319487 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.319561 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.319622 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:04:22.320155 9152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.316754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:46481 (local address 127.3.240.193:36883)
0927 23:04:22.317813 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.317827 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:22.317969 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:22.318289 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:22.318296 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:22.318323 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:22.319103 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:04:22.319106 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:22.319118 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:04:22.319151 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:22.319448 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:22.319452 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:22.319458 (+ 6us) client_negotiation.cc:769] Sending connection context
0927 23:04:22.319611 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 23:04:22.319656 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"mutex_wait_us":49,"thread_start_us":394,"threads_started":1}
I20260927 23:04:22.320199 9153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.316744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:39339 (local address 127.3.240.193:36241)
0927 23:04:22.317544 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.317562 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:22.317737 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:22.318069 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:22.318073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:22.318097 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:22.318871 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:04:22.318877 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:22.318897 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:04:22.318949 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:22.319379 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:22.319383 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:22.319390 (+ 7us) client_negotiation.cc:769] Sending connection context
0927 23:04:22.319535 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 23:04:22.319629 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":702,"threads_started":1}
I20260927 23:04:22.321161 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c863665461ed4466bb8b722998526552" candidate_uuid: "e7c21015921b447e86c432cde4b727eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" is_pre_election: true
I20260927 23:04:22.321164 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c863665461ed4466bb8b722998526552" candidate_uuid: "e7c21015921b447e86c432cde4b727eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" is_pre_election: true
I20260927 23:04:22.321646 8950 raft_consensus.cc:2454] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c21015921b447e86c432cde4b727eb in term 0.
I20260927 23:04:22.321646 9080 raft_consensus.cc:2454] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c21015921b447e86c432cde4b727eb in term 0.
I20260927 23:04:22.322329 8755 leader_election.cc:302] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b17e185b1c4c4c748f1a5d8c1cba6fc8, e7c21015921b447e86c432cde4b727eb; no voters:
I20260927 23:04:22.322767 9151 raft_consensus.cc:2781] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:22.322922 9151 raft_consensus.cc:482] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:22.323051 9151 raft_consensus.cc:3037] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:22.325110 9151 raft_consensus.cc:504] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.325688 9151 leader_election.cc:288] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [CANDIDATE]: Term 1 election: Requested vote from peers b17e185b1c4c4c748f1a5d8c1cba6fc8 (127.3.240.194:39339), 46ccbdaeada3465ba8cad384b3d81ea2 (127.3.240.195:46481)
I20260927 23:04:22.326169 8950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c863665461ed4466bb8b722998526552" candidate_uuid: "e7c21015921b447e86c432cde4b727eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8"
I20260927 23:04:22.326169 9080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c863665461ed4466bb8b722998526552" candidate_uuid: "e7c21015921b447e86c432cde4b727eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46ccbdaeada3465ba8cad384b3d81ea2"
I20260927 23:04:22.326362 9080 raft_consensus.cc:3037] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:22.326362 8950 raft_consensus.cc:3037] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:22.329447 8950 raft_consensus.cc:2454] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c21015921b447e86c432cde4b727eb in term 1.
I20260927 23:04:22.329447 9080 raft_consensus.cc:2454] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c21015921b447e86c432cde4b727eb in term 1.
I20260927 23:04:22.329833 8755 leader_election.cc:302] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 46ccbdaeada3465ba8cad384b3d81ea2, b17e185b1c4c4c748f1a5d8c1cba6fc8, e7c21015921b447e86c432cde4b727eb; no voters:
I20260927 23:04:22.330155 9151 raft_consensus.cc:2781] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:22.330561 9151 raft_consensus.cc:686] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 1 LEADER]: Becoming Leader. State: Replica: e7c21015921b447e86c432cde4b727eb, State: Running, Role: LEADER
I20260927 23:04:22.331043 9151 consensus_queue.cc:237] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } }
I20260927 23:04:22.335376 8679 catalog_manager.cc:5495] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb reported cstate change: term changed from 0 to 1, leader changed from <none> to e7c21015921b447e86c432cde4b727eb (127.3.240.193). New cstate: current_term: 1 leader_uuid: "e7c21015921b447e86c432cde4b727eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c21015921b447e86c432cde4b727eb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 } health_report { overall_health: UNKNOWN } } }
I20260927 23:04:22.415634 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.413434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.3.240.193:35391)
0927 23:04:22.413620 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.413625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.413645 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:04:22.413792 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.413796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.413871 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.413985 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.414901 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.414906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.414911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.414941 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.414975 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.415061 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.415210 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.415342 (+ 132us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.415393 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:04:22.423471 8950 raft_consensus.cc:1261] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 1 FOLLOWER]: Refusing update from remote peer e7c21015921b447e86c432cde4b727eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:22.423472 9080 raft_consensus.cc:1261] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 1 FOLLOWER]: Refusing update from remote peer e7c21015921b447e86c432cde4b727eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:22.424458 9154 consensus_queue.cc:1035] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "46ccbdaeada3465ba8cad384b3d81ea2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:22.424778 9151 consensus_queue.cc:1035] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b17e185b1c4c4c748f1a5d8c1cba6fc8" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:22.449103 8679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:22.449357 8679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:22.452901 8679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f9ec92d02c44514b8fef81af15f6f09: Sending DeleteTablet for 3 replicas of tablet c863665461ed4466bb8b722998526552
I20260927 23:04:22.453979 9060 tablet_service.cc:1514] Processing DeleteTablet for tablet c863665461ed4466bb8b722998526552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:22 UTC) from {username='slave'} at 127.0.0.1:42344
I20260927 23:04:22.454257 8930 tablet_service.cc:1514] Processing DeleteTablet for tablet c863665461ed4466bb8b722998526552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:22 UTC) from {username='slave'} at 127.0.0.1:44876
I20260927 23:04:22.454749 9167 tablet_replica.cc:330] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: stopping tablet replica
I20260927 23:04:22.454684 8800 tablet_service.cc:1514] Processing DeleteTablet for tablet c863665461ed4466bb8b722998526552 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:22 UTC) from {username='slave'} at 127.0.0.1:56082
I20260927 23:04:22.455119 9167 raft_consensus.cc:2229] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:22.455622 9167 raft_consensus.cc:2258] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:22.457584 9167 ts_tablet_manager.cc:1905] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:22.460742 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 8738
I20260927 23:04:22.460860 9169 tablet_replica.cc:330] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: stopping tablet replica
I20260927 23:04:22.461284 9169 raft_consensus.cc:2229] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:22.461877 9169 raft_consensus.cc:2258] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:22.464620 9167 ts_tablet_manager.cc:1918] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:22.464795 9167 log.cc:1196] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/c863665461ed4466bb8b722998526552
I20260927 23:04:22.465297 9167 ts_tablet_manager.cc:1939] T c863665461ed4466bb8b722998526552 P 46ccbdaeada3465ba8cad384b3d81ea2: Deleting consensus metadata
I20260927 23:04:22.466976 8666 catalog_manager.cc:4841] TS 46ccbdaeada3465ba8cad384b3d81ea2 (127.3.240.195:46481): tablet c863665461ed4466bb8b722998526552 (table test-table [id=a689a350b95c4335a62156f2ad961c73]) successfully deleted
I20260927 23:04:22.468883 9168 tablet_replica.cc:330] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: stopping tablet replica
I20260927 23:04:22.469010 9169 ts_tablet_manager.cc:1905] T c863665461ed4466bb8b722998526552 P e7c21015921b447e86c432cde4b727eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:22.469578 9168 raft_consensus.cc:2229] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:22.470088 9168 raft_consensus.cc:2258] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:22.470520 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.462637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56088 (local address 127.3.240.193:35391)
0927 23:04:22.462878 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.462887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.468323 (+ 5436us) server_negotiation.cc:407] Connection header received
0927 23:04:22.468494 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.468500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.468580 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.468673 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.469619 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.469624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.469629 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.469645 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.469679 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.469693 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.469795 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.470164 (+ 369us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.470216 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 23:04:22.471441 8860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:22.472018 9168 ts_tablet_manager.cc:1905] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:22.476954 9168 ts_tablet_manager.cc:1918] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:22.477098 9168 log.cc:1196] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/c863665461ed4466bb8b722998526552
I20260927 23:04:22.477571 9168 ts_tablet_manager.cc:1939] T c863665461ed4466bb8b722998526552 P b17e185b1c4c4c748f1a5d8c1cba6fc8: Deleting consensus metadata
I20260927 23:04:22.479138 8665 catalog_manager.cc:4841] TS b17e185b1c4c4c748f1a5d8c1cba6fc8 (127.3.240.194:39339): tablet c863665461ed4466bb8b722998526552 (table test-table [id=a689a350b95c4335a62156f2ad961c73]) successfully deleted
I20260927 23:04:22.564311 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 8738
W20260927 23:04:22.581597 8665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:35391 (error 108)
I20260927 23:04:22.581664 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 8868
I20260927 23:04:22.583580 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.582815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35391 (local address 127.0.0.1:56098)
0927 23:04:22.583223 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.583371 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":248,"threads_started":1}
W20260927 23:04:22.583817 8665 catalog_manager.cc:4584] TS e7c21015921b447e86c432cde4b727eb (127.3.240.193:35391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c863665461ed4466bb8b722998526552: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35391: connect: Connection refused (error 111)
I20260927 23:04:22.584613 9142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.582583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.3.240.194:39339)
0927 23:04:22.582762 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.582768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.582790 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:04:22.582855 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.582861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.582966 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.583093 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.584074 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.584080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.584085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.584102 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.584143 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.584155 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.584245 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.584362 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.584402 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:04:22.585193 8990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:22.637305 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.636879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35391 (local address 127.0.0.1:56110)
0927 23:04:22.637077 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.637169 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 23:04:22.638319 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.638009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35391 (local address 127.0.0.1:56112)
0927 23:04:22.638133 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.638198 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260927 23:04:22.667027 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 8868
I20260927 23:04:22.683147 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 8998
I20260927 23:04:22.685881 9141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.683881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.3.240.195:46481)
0927 23:04:22.684057 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:22.684064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:22.684095 (+ 31us) server_negotiation.cc:407] Connection header received
0927 23:04:22.684277 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:22.684282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:22.684364 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:22.684460 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:22.685307 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:22.685313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:22.685317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:22.685333 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:22.685370 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:22.685381 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:22.685502 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:22.685574 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 23:04:22.685630 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:04:22.686483 9120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:22.698778 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.698398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35391 (local address 127.0.0.1:56118)
0927 23:04:22.698575 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.698660 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260927 23:04:22.699824 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.699475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35391 (local address 127.0.0.1:56120)
0927 23:04:22.699603 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.699667 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 23:04:22.779640 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 8998
I20260927 23:04:22.781826 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.781292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35391 (local address 127.0.0.1:56134)
0927 23:04:22.781538 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.781654 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 23:04:22.782996 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:22.782472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35391 (local address 127.0.0.1:56136)
0927 23:04:22.782706 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:22.782802 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 23:04:22.795614 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 8648
I20260927 23:04:22.796569 8709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:22.895891 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 8648
2026-09-27T23:04:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2638 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260927 23:04:22.934816 4035 mini_oidc.cc:152] Starting JWKS server
I20260927 23:04:22.937336 4035 webserver.cc:460] Webserver started at https://127.0.0.1:37375/ using document root <none> and password file <none>
I20260927 23:04:22.937566 4035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 23:04:22.938109 4035 webserver.cc:460] Webserver started at http://127.0.0.1:46149/ using document root <none> and password file <none>
2026-09-27T23:04:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:22Z Disabled control of system clock
I20260927 23:04:22.961597 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43169
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:36729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:43169
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37375/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/testchainca.cert with env {}
W20260927 23:04:23.263657 9182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:23.263968 9182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:23.264139 9182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:23.270819 9182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:23.271008 9182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:23.271095 9182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:23.271198 9182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:23.271279 9182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 23:04:23.271363 9182 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 23:04:23.271451 9182 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37375/jwks.json
I20260927 23:04:23.280929 9182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:43169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37375/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:23.282207 9182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:23.283711 9182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:23.295583 9190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:23.295660 9187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:23.295629 9188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:23.296640 9182 server_base.cc:1034] running on GCE node
I20260927 23:04:23.297441 9182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:23.298705 9182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:23.299961 9182 hybrid_clock.cc:648] HybridClock initialized: now 1790550263299896 us; error 44 us; skew 500 ppm
I20260927 23:04:23.300307 9182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:23.306020 9182 webserver.cc:460] Webserver started at http://127.3.240.254:44317/ using document root <none> and password file <none>
I20260927 23:04:23.306663 9182 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:23.306787 9182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:23.307140 9182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:23.309752 9182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "4b524b2d84c14e03a9cea657a6d67ddd"
format_stamp: "Formatted at 2026-09-27 23:04:23 on dist-test-slave-lmfp"
server_key: "317d5c95b67df16334e00fed32bb5ca8"
server_key_iv: "831e188061262fee05a7a1f30c012861"
server_key_version: "encryptionkey@0"
I20260927 23:04:23.310458 9182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "4b524b2d84c14e03a9cea657a6d67ddd"
format_stamp: "Formatted at 2026-09-27 23:04:23 on dist-test-slave-lmfp"
server_key: "317d5c95b67df16334e00fed32bb5ca8"
server_key_iv: "831e188061262fee05a7a1f30c012861"
server_key_version: "encryptionkey@0"
I20260927 23:04:23.315227 9182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 23:04:23.318365 9196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:23.319779 9182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:23.319989 9182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "4b524b2d84c14e03a9cea657a6d67ddd"
format_stamp: "Formatted at 2026-09-27 23:04:23 on dist-test-slave-lmfp"
server_key: "317d5c95b67df16334e00fed32bb5ca8"
server_key_iv: "831e188061262fee05a7a1f30c012861"
server_key_version: "encryptionkey@0"
I20260927 23:04:23.320202 9182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:23.333117 9182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:23.333828 9182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:23.334137 9182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:23.352535 9182 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:43169
I20260927 23:04:23.352552 9247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:43169 every 8 connection(s)
I20260927 23:04:23.353861 9182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:23.356287 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 9182
I20260927 23:04:23.356485 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:23.357265 4035 external_mini_cluster.cc:1428] Setting key 1b5776bf9c57db491eca25c718917682
I20260927 23:04:23.357904 9248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:23.370734 9248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd: Bootstrap starting.
I20260927 23:04:23.370831 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:23.359231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.3.240.254:43169)
0927 23:04:23.360209 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:23.360219 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:23.360264 (+ 45us) server_negotiation.cc:407] Connection header received
0927 23:04:23.362128 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:23.362147 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:23.362573 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:23.362926 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:23.363536 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:23.364887 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:23.366748 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:23.367441 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:23.367598 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:23.368766 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:23.368773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:23.368786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:23.369271 (+ 485us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:23.369337 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:23.369355 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:23.369433 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:23.369647 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 23:04:23.369945 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":365,"threads_started":1}
I20260927 23:04:23.373729 9248 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:23.374222 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:04:23.374827 9248 log.cc:824] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:23.377363 9248 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd: No bootstrap required, opened a new log
I20260927 23:04:23.384248 9248 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b524b2d84c14e03a9cea657a6d67ddd" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43169 } }
I20260927 23:04:23.384586 9248 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:23.384685 9248 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b524b2d84c14e03a9cea657a6d67ddd, State: Initialized, Role: FOLLOWER
I20260927 23:04:23.385212 9248 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b524b2d84c14e03a9cea657a6d67ddd" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43169 } }
I20260927 23:04:23.385437 9248 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:23.385571 9248 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:23.385735 9248 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:23.388106 9248 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b524b2d84c14e03a9cea657a6d67ddd" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43169 } }
I20260927 23:04:23.388586 9248 leader_election.cc:302] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b524b2d84c14e03a9cea657a6d67ddd; no voters:
I20260927 23:04:23.389107 9248 leader_election.cc:288] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:23.389240 9253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:23.389880 9253 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [term 1 LEADER]: Becoming Leader. State: Replica: 4b524b2d84c14e03a9cea657a6d67ddd, State: Running, Role: LEADER
I20260927 23:04:23.390376 9253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b524b2d84c14e03a9cea657a6d67ddd" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43169 } }
I20260927 23:04:23.391040 9248 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:23.393070 9255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b524b2d84c14e03a9cea657a6d67ddd. Latest consensus state: current_term: 1 leader_uuid: "4b524b2d84c14e03a9cea657a6d67ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b524b2d84c14e03a9cea657a6d67ddd" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43169 } } }
I20260927 23:04:23.393280 9254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b524b2d84c14e03a9cea657a6d67ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b524b2d84c14e03a9cea657a6d67ddd" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43169 } } }
I20260927 23:04:23.393503 9255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:23.393535 9254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:23.395569 9262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:23.399667 9262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:23.408174 9262 catalog_manager.cc:1347] Generated new cluster ID: bb10d5a48f864ad6b270c57e21636dc0
I20260927 23:04:23.408291 9262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:23.416400 9262 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:23.417537 9262 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:23.424773 9262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b524b2d84c14e03a9cea657a6d67ddd: Generated new TSK 0
I20260927 23:04:23.425499 9262 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 23:04:23.460837 9278 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260927 23:04:23.461032 9278 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 23:04:23.462033 9251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:23.441817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37120 (local address 127.3.240.254:43169)
0927 23:04:23.442003 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:23.442009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:23.442454 (+ 445us) server_negotiation.cc:407] Connection header received
0927 23:04:23.442711 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:23.442717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:23.442805 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:23.442946 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0927 23:04:23.443454 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:23.444547 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:23.445845 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:23.446510 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:23.446646 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:23.447139 (+ 493us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 23:04:23.461457 (+ 14318us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 23:04:23.461760 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37120: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 23:04:23.462133 9251 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37120: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260927 23:04:23.463271 9277 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:23.441702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43169 (local address 127.0.0.1:37120)
0927 23:04:23.442359 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:23.442374 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:23.442519 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:23.442980 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:23.442983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:23.443033 (+ 50us) client_negotiation.cc:189] Negotiated authn=JWT
0927 23:04:23.443323 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:23.443330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:23.444673 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:23.444676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:23.445709 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:23.445718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:23.446669 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:23.446673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:23.446787 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:23.446839 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 23:04:23.461688 (+ 14849us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0927 23:04:23.461890 (+ 202us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.240.254:43169: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":357,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.240.254:43169: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37375/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260927 23:04:23.465129 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 9182
I20260927 23:04:23.466045 9243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:23.551513 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 9182
2026-09-27T23:04:23Z chronyd exiting
I20260927 23:04:23.601254 4035 test_util.cc:175] -----------------------------------------------
I20260927 23:04:23.601382 4035 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790550224434615-4035-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (670 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260927 23:04:23.604941 4035 mini_oidc.cc:152] Starting JWKS server
I20260927 23:04:23.606624 4035 webserver.cc:460] Webserver started at https://127.0.0.1:45655/ using document root <none> and password file <none>
I20260927 23:04:23.606839 4035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 23:04:23.607467 4035 webserver.cc:460] Webserver started at http://127.0.0.1:38541/ using document root <none> and password file <none>
2026-09-27T23:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:23Z Disabled control of system clock
I20260927 23:04:23.631996 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46203
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:43467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46203
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45655/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/testchainca.cert with env {}
W20260927 23:04:23.937705 9285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:23.938028 9285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:23.938185 9285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:23.945237 9285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:23.945375 9285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:23.945463 9285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:23.945572 9285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:23.945652 9285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 23:04:23.945739 9285 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 23:04:23.945811 9285 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45655/jwks.json
I20260927 23:04:23.955439 9285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45655/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:23.956713 9285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:23.958037 9285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:23.969551 9291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:23.969511 9293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:23.969583 9290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:23.970160 9285 server_base.cc:1034] running on GCE node
I20260927 23:04:23.971081 9285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:23.972329 9285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:23.973554 9285 hybrid_clock.cc:648] HybridClock initialized: now 1790550263973491 us; error 58 us; skew 500 ppm
I20260927 23:04:23.973924 9285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:23.976177 9285 webserver.cc:460] Webserver started at http://127.3.240.254:37693/ using document root <none> and password file <none>
I20260927 23:04:23.976847 9285 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:23.976965 9285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:23.977267 9285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:23.979960 9285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "68f3a7481697438f8861f746e0fbd79c"
format_stamp: "Formatted at 2026-09-27 23:04:23 on dist-test-slave-lmfp"
server_key: "a5a253adc0750e7930225719ca7ca553"
server_key_iv: "265bd7f5596720b767f88e88321309b4"
server_key_version: "encryptionkey@0"
I20260927 23:04:23.980680 9285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "68f3a7481697438f8861f746e0fbd79c"
format_stamp: "Formatted at 2026-09-27 23:04:23 on dist-test-slave-lmfp"
server_key: "a5a253adc0750e7930225719ca7ca553"
server_key_iv: "265bd7f5596720b767f88e88321309b4"
server_key_version: "encryptionkey@0"
I20260927 23:04:23.985499 9285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:04:23.988871 9299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:23.990218 9285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:04:23.990434 9285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "68f3a7481697438f8861f746e0fbd79c"
format_stamp: "Formatted at 2026-09-27 23:04:23 on dist-test-slave-lmfp"
server_key: "a5a253adc0750e7930225719ca7ca553"
server_key_iv: "265bd7f5596720b767f88e88321309b4"
server_key_version: "encryptionkey@0"
I20260927 23:04:23.990653 9285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:24.007308 9285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:24.008144 9285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:24.008433 9285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:24.026484 9285 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46203
I20260927 23:04:24.026502 9350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46203 every 8 connection(s)
I20260927 23:04:24.027937 9285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:24.031476 9351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:24.037356 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 9285
I20260927 23:04:24.037555 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:24.038170 4035 external_mini_cluster.cc:1428] Setting key 8f887987ea5f24531a087d33e0568f79
I20260927 23:04:24.041409 9351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c: Bootstrap starting.
I20260927 23:04:24.044665 9351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:24.045835 9351 log.cc:824] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:24.048692 9351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c: No bootstrap required, opened a new log
I20260927 23:04:24.051256 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:24.039877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46242 (local address 127.3.240.254:46203)
0927 23:04:24.040837 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:24.040847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:24.040885 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:04:24.042611 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:24.042648 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:24.043154 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:24.043466 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:24.044021 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.045403 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.047068 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.047684 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.047808 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:24.048969 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:24.048976 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:24.048989 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:24.049496 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:24.049564 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:24.049588 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:24.049677 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:24.049895 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 23:04:24.050238 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":373,"threads_started":1}
I20260927 23:04:24.054878 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:04:24.056758 9351 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f3a7481697438f8861f746e0fbd79c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46203 } }
I20260927 23:04:24.057209 9351 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:24.057323 9351 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f3a7481697438f8861f746e0fbd79c, State: Initialized, Role: FOLLOWER
I20260927 23:04:24.057875 9351 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f3a7481697438f8861f746e0fbd79c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46203 } }
I20260927 23:04:24.058100 9351 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:24.058238 9351 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:24.058406 9351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:24.060451 9351 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f3a7481697438f8861f746e0fbd79c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46203 } }
I20260927 23:04:24.060927 9351 leader_election.cc:302] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f3a7481697438f8861f746e0fbd79c; no voters:
I20260927 23:04:24.061589 9351 leader_election.cc:288] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:24.061749 9356 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:24.062896 9356 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [term 1 LEADER]: Becoming Leader. State: Replica: 68f3a7481697438f8861f746e0fbd79c, State: Running, Role: LEADER
I20260927 23:04:24.063522 9356 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f3a7481697438f8861f746e0fbd79c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46203 } }
I20260927 23:04:24.063781 9351 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:24.066392 9358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f3a7481697438f8861f746e0fbd79c. Latest consensus state: current_term: 1 leader_uuid: "68f3a7481697438f8861f746e0fbd79c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f3a7481697438f8861f746e0fbd79c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46203 } } }
I20260927 23:04:24.066720 9357 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f3a7481697438f8861f746e0fbd79c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f3a7481697438f8861f746e0fbd79c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46203 } } }
I20260927 23:04:24.067013 9357 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:24.066860 9358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:24.068712 9365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:24.073565 9365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:24.081634 9365 catalog_manager.cc:1347] Generated new cluster ID: bb47ceb7fa8d4b7aa353f23e5f989c8e
I20260927 23:04:24.081740 9365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:24.100313 9365 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:24.102231 9365 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:24.108021 9365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f3a7481697438f8861f746e0fbd79c: Generated new TSK 0
I20260927 23:04:24.108760 9365 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260927 23:04:24.141268 9381 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260927 23:04:24.141434 9381 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 23:04:24.142601 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:24.122612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.3.240.254:46203)
0927 23:04:24.122812 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:24.122819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:24.123203 (+ 384us) server_negotiation.cc:407] Connection header received
0927 23:04:24.123408 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:24.123413 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:24.123523 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:24.123682 (+ 159us) server_negotiation.cc:226] Negotiated authn=JWT
0927 23:04:24.124092 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.125219 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.126577 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.127327 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.127464 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:24.127971 (+ 507us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 23:04:24.141911 (+ 13940us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 23:04:24.142298 (+ 387us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":74}
W20260927 23:04:24.142833 9380 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:24.122467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46203 (local address 127.0.0.1:46248)
0927 23:04:24.123086 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:24.123106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:24.123250 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:24.123700 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:24.123703 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:24.123712 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 23:04:24.123935 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:24.123942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.125363 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.125367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:24.126427 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:24.126437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.127528 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.127533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:24.127670 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:24.127702 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 23:04:24.142166 (+ 14464us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0927 23:04:24.142384 (+ 218us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.240.254:46203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":331,"threads_started":1}
W20260927 23:04:24.142833 9354 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46248: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45655/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260927 23:04:24.144073 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 9285
I20260927 23:04:24.145504 9346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:24.236290 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 9285
2026-09-27T23:04:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (684 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260927 23:04:24.289865 4035 mini_oidc.cc:152] Starting JWKS server
I20260927 23:04:24.291656 4035 webserver.cc:460] Webserver started at https://127.0.0.1:37731/ using document root <none> and password file <none>
I20260927 23:04:24.291882 4035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260927 23:04:24.292480 4035 webserver.cc:460] Webserver started at http://127.0.0.1:41141/ using document root <none> and password file <none>
2026-09-27T23:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:24Z Disabled control of system clock
I20260927 23:04:24.316331 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:33929
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:46079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:33929
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37731/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 23:04:24.629514 9388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:24.629849 9388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:24.630015 9388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:24.637187 9388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:24.637326 9388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:24.637419 9388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:24.637534 9388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:24.637614 9388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 23:04:24.637712 9388 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 23:04:24.637799 9388 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37731/jwks.json
I20260927 23:04:24.646803 9388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:33929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:33929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37731/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:24.648080 9388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:24.649850 9388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:24.661543 9394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:24.661557 9393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:24.661527 9396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:24.662447 9388 server_base.cc:1034] running on GCE node
I20260927 23:04:24.663244 9388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:24.664541 9388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:24.665769 9388 hybrid_clock.cc:648] HybridClock initialized: now 1790550264665725 us; error 42 us; skew 500 ppm
I20260927 23:04:24.666181 9388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:24.668447 9388 webserver.cc:460] Webserver started at http://127.3.240.254:42983/ using document root <none> and password file <none>
I20260927 23:04:24.669144 9388 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:24.669253 9388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:24.669585 9388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:24.672376 9388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "987d5a0da778441f84e7af62f0297b06"
format_stamp: "Formatted at 2026-09-27 23:04:24 on dist-test-slave-lmfp"
server_key: "ec994d9ceb01099d9167b8df1a843273"
server_key_iv: "00041d3c54f8ee85cbc3e8882c9d0aa1"
server_key_version: "encryptionkey@0"
I20260927 23:04:24.673221 9388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "987d5a0da778441f84e7af62f0297b06"
format_stamp: "Formatted at 2026-09-27 23:04:24 on dist-test-slave-lmfp"
server_key: "ec994d9ceb01099d9167b8df1a843273"
server_key_iv: "00041d3c54f8ee85cbc3e8882c9d0aa1"
server_key_version: "encryptionkey@0"
I20260927 23:04:24.678201 9388 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 23:04:24.681394 9402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:24.682647 9388 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 23:04:24.682832 9388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "987d5a0da778441f84e7af62f0297b06"
format_stamp: "Formatted at 2026-09-27 23:04:24 on dist-test-slave-lmfp"
server_key: "ec994d9ceb01099d9167b8df1a843273"
server_key_iv: "00041d3c54f8ee85cbc3e8882c9d0aa1"
server_key_version: "encryptionkey@0"
I20260927 23:04:24.683115 9388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:24.721494 9388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:24.722328 9388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:24.722644 9388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:24.740046 9388 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:33929
I20260927 23:04:24.740068 9453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:33929 every 8 connection(s)
I20260927 23:04:24.741343 9388 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:24.742522 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 9388
I20260927 23:04:24.742765 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:24.743577 4035 external_mini_cluster.cc:1428] Setting key c6b367b6c12b23b7bb4d92f530ae1859
I20260927 23:04:24.745985 9454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:24.755944 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:24.745386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.3.240.254:33929)
0927 23:04:24.746380 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:24.746391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:24.746429 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:04:24.748002 (+ 1573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:24.748022 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:24.748458 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:24.748856 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:24.749376 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.750613 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.751873 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.752534 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.752739 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:24.753860 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:24.753866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:24.753878 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:24.754392 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:24.754474 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:24.754491 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:24.754627 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:24.754833 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 23:04:24.755150 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":337,"threads_started":1}
I20260927 23:04:24.756244 9454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06: Bootstrap starting.
I20260927 23:04:24.759370 9454 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:24.759394 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:04:24.760295 9454 log.cc:824] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:24.763060 9454 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06: No bootstrap required, opened a new log
I20260927 23:04:24.769843 9454 raft_consensus.cc:348] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987d5a0da778441f84e7af62f0297b06" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } }
I20260927 23:04:24.770185 9454 raft_consensus.cc:374] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:24.770290 9454 raft_consensus.cc:729] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987d5a0da778441f84e7af62f0297b06, State: Initialized, Role: FOLLOWER
I20260927 23:04:24.770859 9454 consensus_queue.cc:260] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987d5a0da778441f84e7af62f0297b06" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } }
I20260927 23:04:24.771106 9454 raft_consensus.cc:388] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:24.771240 9454 raft_consensus.cc:482] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:24.771413 9454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:24.773329 9454 raft_consensus.cc:504] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987d5a0da778441f84e7af62f0297b06" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } }
I20260927 23:04:24.773777 9454 leader_election.cc:302] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 987d5a0da778441f84e7af62f0297b06; no voters:
I20260927 23:04:24.774315 9454 leader_election.cc:288] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:24.774406 9459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:24.775731 9459 raft_consensus.cc:686] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [term 1 LEADER]: Becoming Leader. State: Replica: 987d5a0da778441f84e7af62f0297b06, State: Running, Role: LEADER
I20260927 23:04:24.776187 9459 consensus_queue.cc:237] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987d5a0da778441f84e7af62f0297b06" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } }
I20260927 23:04:24.776443 9454 sys_catalog.cc:564] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:24.778715 9461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987d5a0da778441f84e7af62f0297b06. Latest consensus state: current_term: 1 leader_uuid: "987d5a0da778441f84e7af62f0297b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987d5a0da778441f84e7af62f0297b06" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } } }
I20260927 23:04:24.779106 9461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:24.778811 9460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "987d5a0da778441f84e7af62f0297b06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987d5a0da778441f84e7af62f0297b06" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } } }
I20260927 23:04:24.779237 9460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06 [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:24.780572 9465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:24.785059 9465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:24.793182 9465 catalog_manager.cc:1347] Generated new cluster ID: 2e7a92217f7b41d8a669de18248d1b36
I20260927 23:04:24.793283 9465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:24.801466 9465 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:24.802534 9465 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:24.812042 9465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 987d5a0da778441f84e7af62f0297b06: Generated new TSK 0
I20260927 23:04:24.812788 9465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:04:24.839226 9457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:24.831460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.3.240.254:33929)
0927 23:04:24.831655 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:24.831661 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:24.832035 (+ 374us) server_negotiation.cc:407] Connection header received
0927 23:04:24.832229 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:24.832234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:24.832322 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:24.832474 (+ 152us) server_negotiation.cc:226] Negotiated authn=JWT
0927 23:04:24.832866 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.833949 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.835290 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.835949 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.836080 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:24.836493 (+ 413us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 23:04:24.838614 (+ 2121us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 23:04:24.838913 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51536: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260927 23:04:24.839339 9457 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51536: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260927 23:04:24.840812 9483 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:24.831324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:33929 (local address 127.0.0.1:51536)
0927 23:04:24.831935 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:24.831949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:24.832063 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:24.832485 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:24.832488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:24.832497 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 23:04:24.832731 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:24.832738 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.834070 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.834073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:24.835145 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:24.835155 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:24.836106 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:24.836110 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:24.836220 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:24.836252 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 23:04:24.838796 (+ 2544us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0927 23:04:24.840418 (+ 1622us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.240.254:33929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37731/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":1340,"thread_start_us":352,"threads_started":1}
I20260927 23:04:24.841804 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 9388
I20260927 23:04:24.842952 9449 generic_service.cc:196] Checking for leaks (request via RPC)
E20260927 23:04:24.850247 9484 webserver.cc:533] Webserver: sslize failed:
E20260927 23:04:24.850353 9484 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260927 23:04:24.929134 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 9388
2026-09-27T23:04:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (692 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-27T23:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:24Z Disabled control of system clock
I20260927 23:04:25.006110 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41981
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:44555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:41981
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/testchainca.cert with env {}
W20260927 23:04:25.316229 9489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:25.316514 9489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:25.316682 9489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:25.323544 9489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:25.323642 9489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:25.323715 9489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:25.323822 9489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:25.323908 9489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260927 23:04:25.323992 9489 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260927 23:04:25.324060 9489 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260927 23:04:25.333343 9489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:41981
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:25.334594 9489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:25.335901 9489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:25.347360 9497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:25.347424 9495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:25.347409 9494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:25.348116 9489 server_base.cc:1034] running on GCE node
I20260927 23:04:25.348768 9489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:25.350037 9489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:25.351251 9489 hybrid_clock.cc:648] HybridClock initialized: now 1790550265351214 us; error 46 us; skew 500 ppm
I20260927 23:04:25.351572 9489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:25.353542 9489 webserver.cc:460] Webserver started at http://127.3.240.254:39379/ using document root <none> and password file <none>
I20260927 23:04:25.354183 9489 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:25.354310 9489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:25.354609 9489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:25.357360 9489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "7fe36b37eef645cb80972348fbf16276"
format_stamp: "Formatted at 2026-09-27 23:04:25 on dist-test-slave-lmfp"
server_key: "fb86f11cbb2359e20d93978a23f185cb"
server_key_iv: "3dc5e2a5123b7f5f724be0083ac7a8e1"
server_key_version: "encryptionkey@0"
I20260927 23:04:25.358163 9489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "7fe36b37eef645cb80972348fbf16276"
format_stamp: "Formatted at 2026-09-27 23:04:25 on dist-test-slave-lmfp"
server_key: "fb86f11cbb2359e20d93978a23f185cb"
server_key_iv: "3dc5e2a5123b7f5f724be0083ac7a8e1"
server_key_version: "encryptionkey@0"
I20260927 23:04:25.362973 9489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 23:04:25.366180 9503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:25.367533 9489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 23:04:25.367712 9489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "7fe36b37eef645cb80972348fbf16276"
format_stamp: "Formatted at 2026-09-27 23:04:25 on dist-test-slave-lmfp"
server_key: "fb86f11cbb2359e20d93978a23f185cb"
server_key_iv: "3dc5e2a5123b7f5f724be0083ac7a8e1"
server_key_version: "encryptionkey@0"
I20260927 23:04:25.367920 9489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:25.393570 9489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:25.394284 9489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:25.394536 9489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:25.412384 9489 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:41981
I20260927 23:04:25.412413 9554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:41981 every 8 connection(s)
I20260927 23:04:25.413610 9489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:25.416985 9555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:25.421764 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 9489
I20260927 23:04:25.421953 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:25.422524 4035 external_mini_cluster.cc:1428] Setting key d1acdb36910973c827b9bda009dbafe1
I20260927 23:04:25.426751 9555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276: Bootstrap starting.
I20260927 23:04:25.429805 9555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:25.430874 9555 log.cc:824] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:25.433830 9555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276: No bootstrap required, opened a new log
I20260927 23:04:25.435508 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:25.424470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43628 (local address 127.3.240.254:41981)
0927 23:04:25.425437 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:25.425446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:25.425482 (+ 36us) server_negotiation.cc:407] Connection header received
0927 23:04:25.426947 (+ 1465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:25.426972 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:25.427423 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:25.427714 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:25.428269 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:25.429482 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:25.431147 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:25.432202 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:25.432322 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:25.433375 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:25.433382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:25.433394 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:04:25.433936 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:25.434013 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:04:25.434032 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:25.434107 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:25.434336 (+ 229us) server_negotiation.cc:299] Negotiation successful
0927 23:04:25.434623 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":397,"threads_started":1}
I20260927 23:04:25.439040 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:04:25.441699 9555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe36b37eef645cb80972348fbf16276" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41981 } }
I20260927 23:04:25.442097 9555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:25.442211 9555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe36b37eef645cb80972348fbf16276, State: Initialized, Role: FOLLOWER
I20260927 23:04:25.442714 9555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe36b37eef645cb80972348fbf16276" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41981 } }
I20260927 23:04:25.442893 9555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:25.442981 9555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:25.443172 9555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:25.445106 9555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe36b37eef645cb80972348fbf16276" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41981 } }
I20260927 23:04:25.445631 9555 leader_election.cc:302] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fe36b37eef645cb80972348fbf16276; no voters:
I20260927 23:04:25.446285 9555 leader_election.cc:288] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:25.446403 9560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:25.447067 9560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe36b37eef645cb80972348fbf16276, State: Running, Role: LEADER
I20260927 23:04:25.447579 9560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe36b37eef645cb80972348fbf16276" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41981 } }
I20260927 23:04:25.448272 9555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:25.450119 9561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fe36b37eef645cb80972348fbf16276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe36b37eef645cb80972348fbf16276" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41981 } } }
I20260927 23:04:25.450445 9561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:25.451663 9568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:25.451499 9562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fe36b37eef645cb80972348fbf16276. Latest consensus state: current_term: 1 leader_uuid: "7fe36b37eef645cb80972348fbf16276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe36b37eef645cb80972348fbf16276" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41981 } } }
I20260927 23:04:25.452716 9562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276 [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:25.457124 9568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:25.465859 9568 catalog_manager.cc:1347] Generated new cluster ID: f93127f961d5462dbd830a57c7f74798
I20260927 23:04:25.465976 9568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:25.489524 9568 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:25.491364 9568 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:25.504112 9568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fe36b37eef645cb80972348fbf16276: Generated new TSK 0
I20260927 23:04:25.505100 9568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:04:25.948019 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:25.518443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.3.240.254:41981)
0927 23:04:25.518598 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:25.518605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:25.518897 (+ 292us) server_negotiation.cc:407] Connection header received
0927 23:04:25.519087 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:25.519093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:25.519168 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:25.519319 (+ 151us) server_negotiation.cc:226] Negotiated authn=JWT
0927 23:04:25.519692 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:25.520756 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:25.522012 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:25.522655 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:25.522795 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:25.523342 (+ 547us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0927 23:04:25.947274 (+423932us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 23:04:25.947703 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43634: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 23:04:25.948170 9558 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43634: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260927 23:04:25.948256 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:25.518318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:41981 (local address 127.0.0.1:43634)
0927 23:04:25.518802 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:25.518815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:25.518921 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:25.519325 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:25.519328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:25.519337 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0927 23:04:25.519564 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:25.519570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:25.520911 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:25.520915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:25.521881 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:25.521899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:25.522869 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:25.522874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:25.523076 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:25.523109 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0927 23:04:25.947593 (+424484us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0927 23:04:25.947859 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.240.254:41981: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":276,"threads_started":1}
I20260927 23:04:25.949397 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 9489
I20260927 23:04:25.950481 9550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:26.047812 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 9489
2026-09-27T23:04:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1100 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (383 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (390 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:26 dist-test-slave-lmfp krb5kdc[9599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:26 dist-test-slave-lmfp krb5kdc[9599](info): set up 2 sockets
Sep 27 23:04:26 dist-test-slave-lmfp krb5kdc[9599](info): commencing operation
krb5kdc: starting...
W20260927 23:04:27.871852 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:27 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:04:28.070344 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36277
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:36277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:28.385669 9612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:28.386009 9612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:28.386173 9612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:28.393302 9612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:28.393445 9612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:28.393570 9612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:28.393683 9612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:28.393761 9612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:28.403440 9612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:36277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:28.404775 9612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:28.406065 9612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:28.417820 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:28.417889 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:28.417775 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:28.418784 9612 server_base.cc:1034] running on GCE node
I20260927 23:04:28.419524 9612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:28.420822 9612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:28.422037 9612 hybrid_clock.cc:648] HybridClock initialized: now 1790550268421994 us; error 28 us; skew 500 ppm
I20260927 23:04:28.422317 9612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:28 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550268, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:28.428517 9612 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:04:28.430119 9612 webserver.cc:460] Webserver started at http://127.3.240.254:40017/ using document root <none> and password file <none>
I20260927 23:04:28.430815 9612 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:28.430941 9612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:28.431299 9612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:28.434206 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "603f68321a9849ebb135af04ad5e9b0b"
format_stamp: "Formatted at 2026-09-27 23:04:28 on dist-test-slave-lmfp"
server_key: "1cf30dd0daa77aa1266ca5228d902045"
server_key_iv: "6ea2e86c3cdd4e658f593b04bf15d178"
server_key_version: "encryptionkey@0"
I20260927 23:04:28.434950 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "603f68321a9849ebb135af04ad5e9b0b"
format_stamp: "Formatted at 2026-09-27 23:04:28 on dist-test-slave-lmfp"
server_key: "1cf30dd0daa77aa1266ca5228d902045"
server_key_iv: "6ea2e86c3cdd4e658f593b04bf15d178"
server_key_version: "encryptionkey@0"
I20260927 23:04:28.440289 9612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260927 23:04:28.443632 9627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:28.444907 9612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:04:28.445086 9612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "603f68321a9849ebb135af04ad5e9b0b"
format_stamp: "Formatted at 2026-09-27 23:04:28 on dist-test-slave-lmfp"
server_key: "1cf30dd0daa77aa1266ca5228d902045"
server_key_iv: "6ea2e86c3cdd4e658f593b04bf15d178"
server_key_version: "encryptionkey@0"
I20260927 23:04:28.445281 9612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:28.458770 9612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:28.462523 9612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:28.462812 9612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:28.481189 9612 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:36277
I20260927 23:04:28.481218 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:36277 every 8 connection(s)
I20260927 23:04:28.482558 9612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:28.486141 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:28.489239 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 9612
I20260927 23:04:28.489442 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:28.490188 4035 external_mini_cluster.cc:1428] Setting key 36d927faf08d508b0c468f08a7ba0a6f
I20260927 23:04:28.498647 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b: Bootstrap starting.
I20260927 23:04:28.502348 9679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:28.503723 9679 log.cc:824] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b: Log is configured to *not* fsync() on all Append() calls
Sep 27 23:04:28 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:28.506678 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b: No bootstrap required, opened a new log
I20260927 23:04:28.512970 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:28.492229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52730 (local address 127.3.240.254:36277)
0927 23:04:28.492957 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:28.492969 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:28.493008 (+ 39us) server_negotiation.cc:407] Connection header received
0927 23:04:28.494602 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:28.494622 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:28.495040 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:28.495382 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:28.497099 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:28.498327 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:28.500070 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:28.500763 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:28.500899 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:28.505781 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:28.505789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:28.505803 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:28.505834 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:28.508950 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:28.509751 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:28.509758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:28.509764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:28.509955 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:28.510356 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:28.510359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:28.510361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:28.510856 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:28.511172 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:28.511567 (+ 395us) server_negotiation.cc:299] Negotiation successful
0927 23:04:28.511963 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":356,"threads_started":1}
I20260927 23:04:28.515553 9679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f68321a9849ebb135af04ad5e9b0b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36277 } }
I20260927 23:04:28.515990 9679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:28.516089 9679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 603f68321a9849ebb135af04ad5e9b0b, State: Initialized, Role: FOLLOWER
I20260927 23:04:28.516628 9679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f68321a9849ebb135af04ad5e9b0b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36277 } }
I20260927 23:04:28.516815 9679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:28.516978 9679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:28.517141 9679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:28.519295 9679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f68321a9849ebb135af04ad5e9b0b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36277 } }
I20260927 23:04:28.519765 9679 leader_election.cc:302] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 603f68321a9849ebb135af04ad5e9b0b; no voters:
I20260927 23:04:28.520328 9679 leader_election.cc:288] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:28.520483 9684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:28.521631 9684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [term 1 LEADER]: Becoming Leader. State: Replica: 603f68321a9849ebb135af04ad5e9b0b, State: Running, Role: LEADER
I20260927 23:04:28.522162 9684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f68321a9849ebb135af04ad5e9b0b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36277 } }
I20260927 23:04:28.522445 9679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:28.525182 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "603f68321a9849ebb135af04ad5e9b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f68321a9849ebb135af04ad5e9b0b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36277 } } }
I20260927 23:04:28.525534 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:28.525272 9686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 603f68321a9849ebb135af04ad5e9b0b. Latest consensus state: current_term: 1 leader_uuid: "603f68321a9849ebb135af04ad5e9b0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "603f68321a9849ebb135af04ad5e9b0b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36277 } } }
I20260927 23:04:28.525741 9686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:28.527570 9691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:28.532619 9691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:28.541571 9691 catalog_manager.cc:1347] Generated new cluster ID: 0f7b27087ed347b0ac5e730bd4ef5bac
I20260927 23:04:28.541695 9691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:28.557816 9691 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:28.559568 9691 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:28.574002 9691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 603f68321a9849ebb135af04ad5e9b0b: Generated new TSK 0
I20260927 23:04:28.575069 9691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:04:28.627398 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36277
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:28.939777 9705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:28.940062 9705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:28.940150 9705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:28.946884 9705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:28.946978 9705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:28.947227 9705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:28.956195 9705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:28.957286 9705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:28.958462 9705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:28.971151 9711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:28.971100 9710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:28.971498 9713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:28.971793 9705 server_base.cc:1034] running on GCE node
I20260927 23:04:28.972357 9705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:28.973184 9705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:28.974391 9705 hybrid_clock.cc:648] HybridClock initialized: now 1790550268974337 us; error 39 us; skew 500 ppm
I20260927 23:04:28.974695 9705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:28 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550268, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:28.977715 9705 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:04:28.979094 9705 webserver.cc:460] Webserver started at http://127.3.240.193:41991/ using document root <none> and password file <none>
I20260927 23:04:28.979686 9705 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:28.979805 9705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:28.980033 9705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:28.982578 9705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "756e34bcce9f483aadb8fbb99da131a3"
format_stamp: "Formatted at 2026-09-27 23:04:28 on dist-test-slave-lmfp"
server_key: "97231da1a59bfe2b5af7c82cef805d30"
server_key_iv: "8f114df15a4673964df351c5e5c14f52"
server_key_version: "encryptionkey@0"
I20260927 23:04:28.983346 9705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "756e34bcce9f483aadb8fbb99da131a3"
format_stamp: "Formatted at 2026-09-27 23:04:28 on dist-test-slave-lmfp"
server_key: "97231da1a59bfe2b5af7c82cef805d30"
server_key_iv: "8f114df15a4673964df351c5e5c14f52"
server_key_version: "encryptionkey@0"
I20260927 23:04:28.988023 9705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 23:04:28.991200 9720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:28.992431 9705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:04:28.992542 9705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "756e34bcce9f483aadb8fbb99da131a3"
format_stamp: "Formatted at 2026-09-27 23:04:28 on dist-test-slave-lmfp"
server_key: "97231da1a59bfe2b5af7c82cef805d30"
server_key_iv: "8f114df15a4673964df351c5e5c14f52"
server_key_version: "encryptionkey@0"
I20260927 23:04:28.992687 9705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:29.022809 9705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:29.027331 9705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:29.027558 9705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:29.028410 9705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:29.029594 9705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:29.029662 9705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:29.029739 9705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:29.029789 9705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:29.062258 9705 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:42595
I20260927 23:04:29.062305 9832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:42595 every 8 connection(s)
I20260927 23:04:29.063573 9705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:29.065963 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 9705
I20260927 23:04:29.066138 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:29.066871 4035 external_mini_cluster.cc:1428] Setting key bd09378b8fb1d40170dde206c5aa771a
Sep 27 23:04:29 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550268, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:29.083738 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:29.066175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:35819 (local address 127.3.240.254:36277)
0927 23:04:29.066663 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:29.066678 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:29.067835 (+ 1157us) server_negotiation.cc:407] Connection header received
0927 23:04:29.069551 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:29.069559 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:29.069664 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:29.069785 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:29.071234 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:29.072414 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:29.073999 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:29.074480 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:29.074571 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:29.078524 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:29.078528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:29.078532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:29.078559 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:29.080861 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:29.081523 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:29.081534 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:29.081537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:29.081635 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:29.082074 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:29.082078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:29.082081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:29.082338 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:29.082501 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:29.083175 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 23:04:29.083407 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":239,"threads_started":1}
I20260927 23:04:29.084621 9836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:29.066779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36277 (local address 127.3.240.193:35819)
0927 23:04:29.067672 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:29.067723 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:29.069306 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:29.069993 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:29.070001 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:29.070459 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:29.071081 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:29.071099 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:29.072566 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:29.072569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:29.073855 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:29.073866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:29.074704 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:29.074711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:29.075074 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:29.075772 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:29.075804 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:29.078322 (+ 2518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:29.081008 (+ 2686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:29.081015 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:29.081029 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:29.081411 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:29.081736 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:29.081741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:29.081745 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:29.081950 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:29.082475 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:29.082482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:29.082804 (+ 322us) client_negotiation.cc:769] Sending connection context
0927 23:04:29.083090 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 23:04:29.083565 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":452,"threads_started":1}
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260927 23:04:29.086125 9833 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36277
I20260927 23:04:29.086404 9833 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:29.087165 9833 heartbeater.cc:511] Master 127.3.240.254:36277 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260927 23:04:29.089219 9644 ts_manager.cc:194] Registered new tserver with Master: 756e34bcce9f483aadb8fbb99da131a3 (127.3.240.193:42595)
I20260927 23:04:29.090580 9644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:35819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:04:29.110646 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36277
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:29.420404 9839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:29.420660 9839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:29.420848 9839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:29.428165 9839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:29.428273 9839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:29.428445 9839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:29.437938 9839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:29.439198 9839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:29.440459 9839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:29.452639 9845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:29.452682 9847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:29.452725 9844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:29.453372 9839 server_base.cc:1034] running on GCE node
I20260927 23:04:29.453998 9839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:29.454900 9839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:29.456151 9839 hybrid_clock.cc:648] HybridClock initialized: now 1790550269456106 us; error 42 us; skew 500 ppm
I20260927 23:04:29.456446 9839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:29 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:29.459501 9839 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:04:29.460835 9839 webserver.cc:460] Webserver started at http://127.3.240.194:36919/ using document root <none> and password file <none>
I20260927 23:04:29.461472 9839 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:29.461609 9839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:29.461920 9839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:29.464748 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "3f43f69002664525aefe825bc707de7b"
format_stamp: "Formatted at 2026-09-27 23:04:29 on dist-test-slave-lmfp"
server_key: "d512a556f11cda409d43dfeeab737346"
server_key_iv: "6fcc5cd96436b7b884b2c1678f0789d3"
server_key_version: "encryptionkey@0"
I20260927 23:04:29.465513 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "3f43f69002664525aefe825bc707de7b"
format_stamp: "Formatted at 2026-09-27 23:04:29 on dist-test-slave-lmfp"
server_key: "d512a556f11cda409d43dfeeab737346"
server_key_iv: "6fcc5cd96436b7b884b2c1678f0789d3"
server_key_version: "encryptionkey@0"
I20260927 23:04:29.470475 9839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 23:04:29.473711 9854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:29.474887 9839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 23:04:29.475080 9839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "3f43f69002664525aefe825bc707de7b"
format_stamp: "Formatted at 2026-09-27 23:04:29 on dist-test-slave-lmfp"
server_key: "d512a556f11cda409d43dfeeab737346"
server_key_iv: "6fcc5cd96436b7b884b2c1678f0789d3"
server_key_version: "encryptionkey@0"
I20260927 23:04:29.475291 9839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:29.488456 9839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:29.493263 9839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:29.493564 9839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:29.494571 9839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:29.496134 9839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:29.496222 9839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:29.496342 9839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:29.496412 9839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:29.528978 9839 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:43487
I20260927 23:04:29.529054 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:43487 every 8 connection(s)
I20260927 23:04:29.530257 9839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:29.537222 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 9839
I20260927 23:04:29.537405 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:29.537977 4035 external_mini_cluster.cc:1428] Setting key ff388f7cdb36f06ab769f5c48159596c
Sep 27 23:04:29 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:29.551352 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:29.533022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:35131 (local address 127.3.240.254:36277)
0927 23:04:29.533215 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:29.533221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:29.534251 (+ 1030us) server_negotiation.cc:407] Connection header received
0927 23:04:29.535962 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:29.535966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:29.536027 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:29.536112 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:29.537564 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:29.538622 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:29.540264 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:29.540957 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:29.541101 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:29.544723 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:29.544728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:29.544731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:29.544758 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:29.548109 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:29.548905 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:29.548913 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:29.548919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:29.549084 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:29.549549 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:29.549557 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:29.549563 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:29.549889 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:29.550047 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:29.550659 (+ 612us) server_negotiation.cc:299] Negotiation successful
0927 23:04:29.550891 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:04:29.551959 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:29.533460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36277 (local address 127.3.240.194:35131)
0927 23:04:29.534105 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:29.534144 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:29.535732 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:29.536297 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:29.536305 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:29.536801 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:29.537323 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:29.537340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:29.538781 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:29.538785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:29.540126 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:29.540137 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:29.541141 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:29.541145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:29.541318 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:29.541966 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:29.541987 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:29.544524 (+ 2537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:29.548252 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:29.548258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:29.548269 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:29.548776 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:29.549200 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:29.549204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:29.549206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:29.549406 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:29.550002 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:29.550007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:29.550316 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 23:04:29.550523 (+ 207us) client_negotiation.cc:240] Negotiation successful
0927 23:04:29.550853 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":259,"threads_started":1}
I20260927 23:04:29.553297 9967 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36277
I20260927 23:04:29.553597 9967 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:29.554515 9967 heartbeater.cc:511] Master 127.3.240.254:36277 requested a full tablet report, sending...
I20260927 23:04:29.555912 9644 ts_manager.cc:194] Registered new tserver with Master: 3f43f69002664525aefe825bc707de7b (127.3.240.194:43487)
I20260927 23:04:29.556694 9644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:35131
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:04:29.580462 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36277
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:43223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:29.905341 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:29.905615 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:29.905815 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:29.913226 9972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:29.913348 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:29.913574 9972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:29.923345 9972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:29.924671 9972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:29.926083 9972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:29.939500 9978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:29.939435 9980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:29.939999 9977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:29.940374 9972 server_base.cc:1034] running on GCE node
I20260927 23:04:29.940907 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:29.941785 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:29.943137 9972 hybrid_clock.cc:648] HybridClock initialized: now 1790550269943073 us; error 37 us; skew 500 ppm
I20260927 23:04:29.943454 9972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:29 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:29.946511 9972 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:04:29.947907 9972 webserver.cc:460] Webserver started at http://127.3.240.195:41817/ using document root <none> and password file <none>
I20260927 23:04:29.948624 9972 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:29.948740 9972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:29.949023 9972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:29.951767 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "1de2355ca380423fb19e94895280da2b"
format_stamp: "Formatted at 2026-09-27 23:04:29 on dist-test-slave-lmfp"
server_key: "d272efc5f09123873abbae080f2a60f0"
server_key_iv: "2106f559db3e6cb6e5dfa1432c2e4d78"
server_key_version: "encryptionkey@0"
I20260927 23:04:29.952435 9972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "1de2355ca380423fb19e94895280da2b"
format_stamp: "Formatted at 2026-09-27 23:04:29 on dist-test-slave-lmfp"
server_key: "d272efc5f09123873abbae080f2a60f0"
server_key_iv: "2106f559db3e6cb6e5dfa1432c2e4d78"
server_key_version: "encryptionkey@0"
I20260927 23:04:29.957110 9972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:04:29.960618 9987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:29.961871 9972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 23:04:29.962059 9972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "1de2355ca380423fb19e94895280da2b"
format_stamp: "Formatted at 2026-09-27 23:04:29 on dist-test-slave-lmfp"
server_key: "d272efc5f09123873abbae080f2a60f0"
server_key_iv: "2106f559db3e6cb6e5dfa1432c2e4d78"
server_key_version: "encryptionkey@0"
I20260927 23:04:29.962256 9972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:29.983644 9972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:29.987599 9972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:29.987854 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:29.988682 9972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:29.989885 9972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:29.989992 9972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:29.990129 9972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:29.990197 9972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:30.022424 9972 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:32905
I20260927 23:04:30.022507 10099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:32905 every 8 connection(s)
I20260927 23:04:30.023867 9972 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:30.027215 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 9972
I20260927 23:04:30.027381 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:30.027771 4035 external_mini_cluster.cc:1428] Setting key f858c5efdabb09ad1091842225004ada
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550269, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:30.044132 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:30.026974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:42763 (local address 127.3.240.254:36277)
0927 23:04:30.027169 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.027175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.028632 (+ 1457us) server_negotiation.cc:407] Connection header received
0927 23:04:30.031086 (+ 2454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.031089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.031153 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.031231 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.032802 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.033560 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.035047 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.035431 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.035526 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:30.038620 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:30.038626 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:30.038630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:30.038664 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:30.040975 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.041826 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.041834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.041838 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.041965 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.042459 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.042464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.042467 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.042861 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:30.043044 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:30.043730 (+ 686us) server_negotiation.cc:299] Negotiation successful
0927 23:04:30.043885 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 23:04:30.045012 10102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:30.027627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36277 (local address 127.3.240.195:42763)
0927 23:04:30.028467 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:30.028513 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:30.030714 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:30.031506 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:30.031514 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:30.032070 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:30.032666 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:30.032682 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.033699 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.033703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:30.034880 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:30.034891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.035605 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.035609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:30.035787 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:30.036379 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:30.036399 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:30.038404 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:30.041120 (+ 2716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:30.041127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:30.041143 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:30.041718 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:30.042094 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:30.042099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:30.042103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:30.042344 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:30.043061 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:30.043068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:30.043413 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 23:04:30.043649 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 23:04:30.044006 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":354,"threads_started":1}
I20260927 23:04:30.046180 10100 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36277
I20260927 23:04:30.046423 10100 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:30.047116 10100 heartbeater.cc:511] Master 127.3.240.254:36277 requested a full tablet report, sending...
I20260927 23:04:30.048398 9644 ts_manager.cc:194] Registered new tserver with Master: 1de2355ca380423fb19e94895280da2b (127.3.240.195:32905)
I20260927 23:04:30.049154 9644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:42763
I20260927 23:04:30.054855 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/test-admin.keytab.
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:30.082336 4035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260927 23:04:30.086891 10107 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:30.084847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.3.240.193:42595)
0927 23:04:30.085400 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.085406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.085429 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:04:30.085491 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.085497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.085676 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.085797 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.086583 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55604: BlockingRecv error: recv got EOF from 127.0.0.1:55604 (error 108)
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":300,"threads_started":1}
W20260927 23:04:30.087262 4035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5cc' of type FILE
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:30.088660 4035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:30.089901 4035 init.cc:295] Successfully reacquired a new kerberos TGT
I20260927 23:04:30.093885 9833 heartbeater.cc:503] Master 127.3.240.254:36277 was elected leader, sending a full tablet report...
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:04:30.104724 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:30.092096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.3.240.193:42595)
0927 23:04:30.092280 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.092285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.092429 (+ 144us) server_negotiation.cc:407] Connection header received
0927 23:04:30.092541 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.092544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.092592 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.092667 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.093541 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.094742 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.096018 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.096522 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.096613 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:30.099810 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:30.099818 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:30.099830 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:30.099856 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:30.102436 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.103139 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.103145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.103147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.103246 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.103572 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.103575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.103577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.103852 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:30.104051 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:30.104254 (+ 203us) server_negotiation.cc:299] Negotiation successful
0927 23:04:30.104480 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 23:04:30.109654 10107 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:30.108226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.3.240.193:42595)
0927 23:04:30.108417 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.108421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.108637 (+ 216us) server_negotiation.cc:407] Connection header received
0927 23:04:30.108742 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.108745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.108791 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.108854 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.109432 (+ 578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55610: BlockingRecv error: recv got EOF from 127.0.0.1:55610 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260927 23:04:30.109761 4035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5cc' of type FILE
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:30.111063 4035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:30.112219 4035 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:04:30.124823 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:30.113332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.3.240.193:42595)
0927 23:04:30.113480 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.113484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.113650 (+ 166us) server_negotiation.cc:407] Connection header received
0927 23:04:30.113823 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.113831 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.113874 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.113976 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.114753 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.115433 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.116525 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.116921 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.117065 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:30.120370 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:30.120374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:30.120378 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:30.120418 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:30.122735 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.123399 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.123403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.123406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.123517 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.123851 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.123854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.123855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.124089 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:30.124271 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:30.124472 (+ 201us) server_negotiation.cc:299] Negotiation successful
0927 23:04:30.124612 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 23:04:30.128963 10107 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:30.127288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.3.240.193:42595)
0927 23:04:30.127510 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.127518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.127747 (+ 229us) server_negotiation.cc:407] Connection header received
0927 23:04:30.127937 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.127940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.128000 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.128096 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.128701 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55626: BlockingRecv error: recv got EOF from 127.0.0.1:55626 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260927 23:04:30.129298 4035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790550224434615-4035-0/krb5kdc/krb5cc' of type FILE
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:30.130776 4035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:30.132100 4035 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:04:30.144937 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:30.133319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.3.240.193:42595)
0927 23:04:30.133515 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.133520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.133771 (+ 251us) server_negotiation.cc:407] Connection header received
0927 23:04:30.133868 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.133871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.133916 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.133992 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.134855 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.135608 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.136745 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.137135 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.137220 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:30.140299 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:30.140303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:30.140306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:30.140330 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:30.142730 (+ 2400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.143367 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.143371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.143373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.143502 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.143807 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.143809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.143811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.144047 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:30.144293 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:30.144435 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 23:04:30.144569 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:04:30.146339 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 9705
I20260927 23:04:30.156837 10107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:30.146958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.3.240.193:42595)
0927 23:04:30.147137 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.147145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.147400 (+ 255us) server_negotiation.cc:407] Connection header received
0927 23:04:30.147508 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.147511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.147555 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.147622 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.148440 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.149029 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.150140 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.150504 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.150593 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:30.152286 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:30.152290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:30.152293 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:30.152319 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:30.154663 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.155428 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.155432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.155434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.155552 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.155840 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.155843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.155845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.156067 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:30.156234 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:30.156491 (+ 257us) server_negotiation.cc:299] Negotiation successful
0927 23:04:30.156618 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 23:04:30.157359 9828 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:30.237866 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 9705
I20260927 23:04:30.254281 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 9839
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:04:30.269408 10120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:30.255128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.3.240.194:43487)
0927 23:04:30.255726 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.255733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.255756 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:04:30.255814 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.255820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.256024 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.256230 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.257223 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.258197 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.259556 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.260295 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.260461 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:30.264001 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:30.264017 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:30.264030 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:30.264063 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:30.266718 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.267464 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.267472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.267474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.267607 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.268004 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.268009 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.268010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.268311 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:30.268602 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:30.268908 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 23:04:30.269150 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":373,"threads_started":1}
I20260927 23:04:30.270576 9962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:30.348237 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 9839
I20260927 23:04:30.365981 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 9972
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[9599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:04:30.380739 10122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:30.366931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.3.240.195:32905)
0927 23:04:30.367776 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:30.367783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:30.367833 (+ 50us) server_negotiation.cc:407] Connection header received
0927 23:04:30.367896 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:30.367903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:30.368122 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:30.368256 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:30.369177 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.370094 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.371377 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:30.371971 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:30.372085 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:30.375872 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:30.375882 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:30.375899 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:30.375953 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:30.378440 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.378947 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.378955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.378957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.379124 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:30.379500 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:30.379504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:30.379506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:30.379812 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:30.380035 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:30.380261 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 23:04:30.380448 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":525,"threads_started":1}
I20260927 23:04:30.382006 10095 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:30.461114 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 9972
I20260927 23:04:30.475684 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 9612
I20260927 23:04:30.476791 9674 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:30.573766 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 9612
2026-09-27T23:04:30Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3755 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[10127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[10127](info): set up 2 sockets
Sep 27 23:04:30 dist-test-slave-lmfp krb5kdc[10127](info): commencing operation
krb5kdc: starting...
W20260927 23:04:31.615540 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:31 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.254.keytab.
Entry for principal oryx/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.254.keytab.
I20260927 23:04:31.802809 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37357
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.254.keytab
--principal=oryx/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:32.103188 10140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:32.103482 10140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:32.103600 10140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:32.110056 10140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:32.110142 10140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:32.110203 10140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:32.110248 10140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:32.110291 10140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:32.119299 10140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.254.keytab
--principal=oryx/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:32.120473 10140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:32.121634 10140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:32.132721 10148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:32.132766 10145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:32.132800 10146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:32.133698 10140 server_base.cc:1034] running on GCE node
I20260927 23:04:32.134374 10140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:32.135778 10140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:32.137008 10140 hybrid_clock.cc:648] HybridClock initialized: now 1790550272136968 us; error 36 us; skew 500 ppm
I20260927 23:04:32.137315 10140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:32 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550272, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:32.140262 10140 init.cc:377] Logged in from keytab as oryx/127.3.240.254@KRBTEST.COM (short username oryx)
I20260927 23:04:32.141808 10140 webserver.cc:460] Webserver started at http://127.3.240.254:34485/ using document root <none> and password file <none>
I20260927 23:04:32.142477 10140 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:32.142575 10140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:32.142836 10140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:32.145658 10140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "a56b7432116b47e8ac17984a7421f31d"
format_stamp: "Formatted at 2026-09-27 23:04:32 on dist-test-slave-lmfp"
server_key: "349de77df28c98b227f94fb815b37cd7"
server_key_iv: "c5e6e4e6e1c87f1cd30ff31d26039fd7"
server_key_version: "encryptionkey@0"
I20260927 23:04:32.146451 10140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "a56b7432116b47e8ac17984a7421f31d"
format_stamp: "Formatted at 2026-09-27 23:04:32 on dist-test-slave-lmfp"
server_key: "349de77df28c98b227f94fb815b37cd7"
server_key_iv: "c5e6e4e6e1c87f1cd30ff31d26039fd7"
server_key_version: "encryptionkey@0"
I20260927 23:04:32.151867 10140 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 23:04:32.154951 10155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:32.156204 10140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:32.156332 10140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "a56b7432116b47e8ac17984a7421f31d"
format_stamp: "Formatted at 2026-09-27 23:04:32 on dist-test-slave-lmfp"
server_key: "349de77df28c98b227f94fb815b37cd7"
server_key_iv: "c5e6e4e6e1c87f1cd30ff31d26039fd7"
server_key_version: "encryptionkey@0"
I20260927 23:04:32.156473 10140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:32.177300 10140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:32.180989 10140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:32.181193 10140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:32.198634 10140 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:37357
I20260927 23:04:32.198671 10206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:37357 every 8 connection(s)
I20260927 23:04:32.199991 10140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:32.200578 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 10140
I20260927 23:04:32.200790 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:32.201574 4035 external_mini_cluster.cc:1428] Setting key 1eb7cd57d8a6b2980dd365923f9956fd
I20260927 23:04:32.204281 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 23:04:32 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:32.216537 10207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: Bootstrap starting.
I20260927 23:04:32.220317 10207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:32.221608 10207 log.cc:824] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:32.223744 10210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:32.203609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45086 (local address 127.3.240.254:37357)
0927 23:04:32.204335 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:32.204346 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:32.204406 (+ 60us) server_negotiation.cc:407] Connection header received
0927 23:04:32.205932 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:32.205952 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:32.206287 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:32.206637 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:32.208331 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:32.209621 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:32.211325 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:32.211919 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:32.212045 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:32.216375 (+ 4330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:32.216394 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:32.216411 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:32.216450 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:32.219984 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:32.220576 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:32.220587 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:32.220594 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:32.220800 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:32.221174 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:32.221181 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:32.221187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:32.221861 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:32.222208 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:32.222483 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 23:04:32.222806 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":348,"threads_started":1}
I20260927 23:04:32.224439 10207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: No bootstrap required, opened a new log
I20260927 23:04:32.231132 10207 raft_consensus.cc:348] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } }
I20260927 23:04:32.231449 10207 raft_consensus.cc:374] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:32.231549 10207 raft_consensus.cc:729] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56b7432116b47e8ac17984a7421f31d, State: Initialized, Role: FOLLOWER
I20260927 23:04:32.232015 10207 consensus_queue.cc:260] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } }
I20260927 23:04:32.232228 10207 raft_consensus.cc:388] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:32.232359 10207 raft_consensus.cc:482] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:32.232518 10207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:32.234488 10207 raft_consensus.cc:504] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } }
I20260927 23:04:32.234939 10207 leader_election.cc:302] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a56b7432116b47e8ac17984a7421f31d; no voters:
I20260927 23:04:32.235582 10207 leader_election.cc:288] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:32.235715 10212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:32.236399 10212 raft_consensus.cc:686] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 1 LEADER]: Becoming Leader. State: Replica: a56b7432116b47e8ac17984a7421f31d, State: Running, Role: LEADER
I20260927 23:04:32.237056 10212 consensus_queue.cc:237] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } }
I20260927 23:04:32.237692 10207 sys_catalog.cc:564] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:32.239725 10213 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } }
I20260927 23:04:32.240044 10213 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:32.240713 10214 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a56b7432116b47e8ac17984a7421f31d. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } }
I20260927 23:04:32.243268 10214 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:32.244453 10219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:32.253717 10219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:32.261427 10219 catalog_manager.cc:1347] Generated new cluster ID: 5d16037075ed46b783798781536f95e9
I20260927 23:04:32.261539 10219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:32.277539 10219 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:32.278755 10219 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:32.293889 10219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: Generated new TSK 0
I20260927 23:04:32.294659 10219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
I20260927 23:04:32.346122 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37357
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:32.662647 10233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:32.662915 10233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:32.663022 10233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:32.670046 10233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:32.670147 10233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:32.670248 10233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:32.679281 10233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:32.680406 10233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:32.681612 10233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:32.693998 10238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:32.693953 10241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:32.694024 10239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:32.694900 10233 server_base.cc:1034] running on GCE node
I20260927 23:04:32.695549 10233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:32.696386 10233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:32.697606 10233 hybrid_clock.cc:648] HybridClock initialized: now 1790550272697571 us; error 47 us; skew 500 ppm
I20260927 23:04:32.697896 10233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:32 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550272, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:32.701030 10233 init.cc:377] Logged in from keytab as oryx/127.3.240.193@KRBTEST.COM (short username oryx)
I20260927 23:04:32.702363 10233 webserver.cc:460] Webserver started at http://127.3.240.193:41267/ using document root <none> and password file <none>
I20260927 23:04:32.703068 10233 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:32.703203 10233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:32.703481 10233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:32.706200 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "e4fd87ed1cbf47b39818dffaa0bc4095"
format_stamp: "Formatted at 2026-09-27 23:04:32 on dist-test-slave-lmfp"
server_key: "26b8ace99dc94f955106d8b89cc0c7e2"
server_key_iv: "ddb07c82a631d101ba3f00bd1df6b151"
server_key_version: "encryptionkey@0"
I20260927 23:04:32.707028 10233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "e4fd87ed1cbf47b39818dffaa0bc4095"
format_stamp: "Formatted at 2026-09-27 23:04:32 on dist-test-slave-lmfp"
server_key: "26b8ace99dc94f955106d8b89cc0c7e2"
server_key_iv: "ddb07c82a631d101ba3f00bd1df6b151"
server_key_version: "encryptionkey@0"
I20260927 23:04:32.711822 10233 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 23:04:32.714980 10248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:32.716391 10233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:04:32.716552 10233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "e4fd87ed1cbf47b39818dffaa0bc4095"
format_stamp: "Formatted at 2026-09-27 23:04:32 on dist-test-slave-lmfp"
server_key: "26b8ace99dc94f955106d8b89cc0c7e2"
server_key_iv: "ddb07c82a631d101ba3f00bd1df6b151"
server_key_version: "encryptionkey@0"
I20260927 23:04:32.716745 10233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:32.743243 10233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:32.747638 10233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:32.747913 10233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:32.748809 10233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:32.750116 10233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:32.750221 10233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:32.750365 10233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:32.750437 10233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:32.781394 10233 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:36617
I20260927 23:04:32.781438 10360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:36617 every 8 connection(s)
I20260927 23:04:32.782588 10233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:32.785789 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 10233
I20260927 23:04:32.785985 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:32.786767 4035 external_mini_cluster.cc:1428] Setting key 0c9286c3b7e365bf7b2cf292b6eaedc8
Sep 27 23:04:32 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550272, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:32.802515 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:32.785147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:44895 (local address 127.3.240.254:37357)
0927 23:04:32.785577 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:32.785585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:32.786703 (+ 1118us) server_negotiation.cc:407] Connection header received
0927 23:04:32.788618 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:32.788622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:32.788697 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:32.788819 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:32.790181 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:32.790867 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:32.792334 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:32.792734 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:32.792915 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:32.797011 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:32.797016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:32.797019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:32.797048 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:32.799554 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:32.800431 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:32.800442 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:32.800444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:32.800557 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:32.801036 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:32.801039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:32.801040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:32.801283 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:32.801390 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:32.802152 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 23:04:32.802287 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":204,"threads_started":1}
I20260927 23:04:32.804123 10364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:32.785667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.193:44895)
0927 23:04:32.786521 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:32.786571 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:32.788370 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:32.789041 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:32.789049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:32.789498 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:32.790027 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:32.790043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:32.791043 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:32.791048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:32.792196 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:32.792208 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:32.792974 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:32.792981 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:32.793239 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:32.793889 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:32.793920 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:32.796799 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:32.799683 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:32.799694 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:32.799712 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:32.800299 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:32.800679 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:32.800685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:32.800692 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:32.800925 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:32.801413 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:32.801422 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:32.801809 (+ 387us) client_negotiation.cc:769] Sending connection context
0927 23:04:32.802092 (+ 283us) client_negotiation.cc:240] Negotiation successful
0927 23:04:32.802590 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":354,"threads_started":1}
I20260927 23:04:32.805629 10361 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37357
I20260927 23:04:32.805900 10361 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260927 23:04:32.806571 10361 heartbeater.cc:511] Master 127.3.240.254:37357 requested a full tablet report, sending...
I20260927 23:04:32.808560 10172 ts_manager.cc:194] Registered new tserver with Master: e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.194@KRBTEST.COM" created.
I20260927 23:04:32.809957 10172 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:44895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
I20260927 23:04:32.831652 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37357
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:33.142329 10367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:33.142596 10367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:33.142709 10367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:33.149411 10367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:33.149500 10367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:33.149603 10367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:33.158710 10367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:33.159983 10367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:33.161350 10367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:33.173796 10372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:33.173879 10373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:33.173926 10375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:33.174731 10367 server_base.cc:1034] running on GCE node
I20260927 23:04:33.175450 10367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:33.176332 10367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:33.177522 10367 hybrid_clock.cc:648] HybridClock initialized: now 1790550273177467 us; error 45 us; skew 500 ppm
I20260927 23:04:33.177825 10367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:33 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550273, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:33.180763 10367 init.cc:377] Logged in from keytab as oryx/127.3.240.194@KRBTEST.COM (short username oryx)
I20260927 23:04:33.182098 10367 webserver.cc:460] Webserver started at http://127.3.240.194:38853/ using document root <none> and password file <none>
I20260927 23:04:33.182782 10367 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:33.182924 10367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:33.183230 10367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:33.185967 10367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "f16ad42ff909452a81b5044bcd3de35d"
format_stamp: "Formatted at 2026-09-27 23:04:33 on dist-test-slave-lmfp"
server_key: "044926f08841fce9e4fb860c868dc277"
server_key_iv: "ff57c10655fd9c8b1d78575f52812809"
server_key_version: "encryptionkey@0"
I20260927 23:04:33.186725 10367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "f16ad42ff909452a81b5044bcd3de35d"
format_stamp: "Formatted at 2026-09-27 23:04:33 on dist-test-slave-lmfp"
server_key: "044926f08841fce9e4fb860c868dc277"
server_key_iv: "ff57c10655fd9c8b1d78575f52812809"
server_key_version: "encryptionkey@0"
I20260927 23:04:33.191634 10367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:04:33.194859 10382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:33.196089 10367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:33.196254 10367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "f16ad42ff909452a81b5044bcd3de35d"
format_stamp: "Formatted at 2026-09-27 23:04:33 on dist-test-slave-lmfp"
server_key: "044926f08841fce9e4fb860c868dc277"
server_key_iv: "ff57c10655fd9c8b1d78575f52812809"
server_key_version: "encryptionkey@0"
I20260927 23:04:33.196436 10367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:33.218178 10367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:33.222534 10367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:33.222769 10367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:33.223867 10367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:33.225030 10367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:33.225098 10367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:33.225178 10367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:33.225219 10367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:33.256487 10367 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:36347
I20260927 23:04:33.256520 10494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:36347 every 8 connection(s)
I20260927 23:04:33.257679 10367 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:33.258306 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 10367
I20260927 23:04:33.258505 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:33.259070 4035 external_mini_cluster.cc:1428] Setting key 2e630cdaa26bd6c3ced1ac26aca7e85d
Sep 27 23:04:33 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550273, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260927 23:04:33.278929 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:33.260343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:50245 (local address 127.3.240.254:37357)
0927 23:04:33.261113 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:33.261119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:33.262235 (+ 1116us) server_negotiation.cc:407] Connection header received
0927 23:04:33.264182 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:33.264190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:33.264277 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:33.264389 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:33.265870 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:33.267053 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:33.268610 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:33.269051 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:33.269204 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:33.272540 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:33.272547 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:33.272552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:33.272590 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:33.275677 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:33.276440 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:33.276447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:33.276454 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:33.276597 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:33.277058 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:33.277064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:33.277067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:33.277343 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:33.277532 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:33.278251 (+ 719us) server_negotiation.cc:299] Negotiation successful
0927 23:04:33.278521 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650}
I20260927 23:04:33.279946 10497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:33.261007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.194:50245)
0927 23:04:33.262029 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:33.262082 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:33.263889 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:33.264598 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:33.264606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:33.265132 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:33.265686 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:33.265705 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:33.267210 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:33.267214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:33.268368 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:33.268379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:33.269305 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:33.269312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:33.269534 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:33.270204 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:33.270228 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:33.272337 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:33.275829 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:33.275839 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:33.275865 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:33.276304 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:33.276716 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:33.276723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:33.276727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:33.276927 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:33.277496 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:33.277504 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:33.277874 (+ 370us) client_negotiation.cc:769] Sending connection context
0927 23:04:33.278160 (+ 286us) client_negotiation.cc:240] Negotiation successful
0927 23:04:33.278693 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":500,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.195@KRBTEST.COM" created.
I20260927 23:04:33.281450 10495 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37357
I20260927 23:04:33.281786 10495 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:33.282745 10495 heartbeater.cc:511] Master 127.3.240.254:37357 requested a full tablet report, sending...
I20260927 23:04:33.284536 10172 ts_manager.cc:194] Registered new tserver with Master: f16ad42ff909452a81b5044bcd3de35d (127.3.240.194:36347)
I20260927 23:04:33.285318 10172 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:50245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
I20260927 23:04:33.305541 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37357
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:33.607300 10500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:33.607543 10500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:33.607708 10500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:33.614415 10500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:33.614521 10500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:33.614689 10500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:33.623911 10500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:33.625092 10500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:33.626332 10500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:33.639169 10505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:33.639294 10506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:33.639424 10508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:33.640203 10500 server_base.cc:1034] running on GCE node
I20260927 23:04:33.640707 10500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:33.641510 10500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:33.642716 10500 hybrid_clock.cc:648] HybridClock initialized: now 1790550273642682 us; error 47 us; skew 500 ppm
I20260927 23:04:33.643052 10500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:33 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550273, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:33.651660 10500 init.cc:377] Logged in from keytab as oryx/127.3.240.195@KRBTEST.COM (short username oryx)
I20260927 23:04:33.653013 10500 webserver.cc:460] Webserver started at http://127.3.240.195:33001/ using document root <none> and password file <none>
I20260927 23:04:33.653637 10500 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:33.653753 10500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:33.654003 10500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:33.656884 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "64ecde76b14f43b09665f78aa1275f04"
format_stamp: "Formatted at 2026-09-27 23:04:33 on dist-test-slave-lmfp"
server_key: "fea633f544d601ee57396405a3e6c1ed"
server_key_iv: "0975d3e631c6da599c6b1cad0fd8be58"
server_key_version: "encryptionkey@0"
I20260927 23:04:33.657598 10500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "64ecde76b14f43b09665f78aa1275f04"
format_stamp: "Formatted at 2026-09-27 23:04:33 on dist-test-slave-lmfp"
server_key: "fea633f544d601ee57396405a3e6c1ed"
server_key_iv: "0975d3e631c6da599c6b1cad0fd8be58"
server_key_version: "encryptionkey@0"
I20260927 23:04:33.663075 10500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 23:04:33.666486 10515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:33.667788 10500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 23:04:33.667969 10500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "64ecde76b14f43b09665f78aa1275f04"
format_stamp: "Formatted at 2026-09-27 23:04:33 on dist-test-slave-lmfp"
server_key: "fea633f544d601ee57396405a3e6c1ed"
server_key_iv: "0975d3e631c6da599c6b1cad0fd8be58"
server_key_version: "encryptionkey@0"
I20260927 23:04:33.668171 10500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:33.684780 10500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:33.688836 10500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:33.689133 10500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:33.690203 10500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:33.691838 10500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:33.691929 10500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:33.692129 10500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:33.692215 10500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:33.727098 10500 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42997
I20260927 23:04:33.727140 10627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42997 every 8 connection(s)
I20260927 23:04:33.728394 10500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:33.733246 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 10500
I20260927 23:04:33.733415 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:33.733798 4035 external_mini_cluster.cc:1428] Setting key d48c19df6efc2bc47d134e2f89ccebc7
Sep 27 23:04:33 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550273, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:33.748178 10363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:33.730906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:42809 (local address 127.3.240.254:37357)
0927 23:04:33.731125 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:33.731139 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:33.732286 (+ 1147us) server_negotiation.cc:407] Connection header received
0927 23:04:33.734248 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:33.734253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:33.734347 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:33.734483 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:33.736239 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:33.737120 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:33.738635 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:33.739141 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:33.739240 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:33.742465 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:33.742470 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:33.742473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:33.742501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:33.745173 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:33.745776 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:33.745780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:33.745782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:33.745892 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:33.746276 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:33.746279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:33.746281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:33.746556 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:33.746668 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:33.747527 (+ 859us) server_negotiation.cc:299] Negotiation successful
0927 23:04:33.747803 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:04:33.748929 10630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:33.731404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.195:42809)
0927 23:04:33.732111 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:33.732160 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:33.733979 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:33.734708 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:33.734719 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:33.735330 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:33.736076 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:33.736100 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:33.737280 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:33.737284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:33.738516 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:33.738526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:33.739331 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:33.739336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:33.739523 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:33.740190 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:33.740215 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:33.742282 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:33.745281 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:33.745286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:33.745296 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:33.745670 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:33.746013 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:33.746016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:33.746018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:33.746172 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:33.746653 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:33.746658 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:33.747005 (+ 347us) client_negotiation.cc:769] Sending connection context
0927 23:04:33.747274 (+ 269us) client_negotiation.cc:240] Negotiation successful
0927 23:04:33.747700 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":263,"threads_started":1}
I20260927 23:04:33.750157 10628 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37357
I20260927 23:04:33.750404 10628 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:33.751114 10628 heartbeater.cc:511] Master 127.3.240.254:37357 requested a full tablet report, sending...
I20260927 23:04:33.752346 10172 ts_manager.cc:194] Registered new tserver with Master: 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997)
I20260927 23:04:33.753026 10172 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:42809
I20260927 23:04:33.760839 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.240.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.253.keytab.
Entry for principal oryx/127.3.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.253.keytab.
I20260927 23:04:33.804227 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:34805
--webserver_interface=127.3.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.253.keytab
--principal=oryx/127.3.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37357,127.3.240.253:34805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:04:33.812729 10361 heartbeater.cc:503] Master 127.3.240.254:37357 was elected leader, sending a full tablet report...
W20260927 23:04:34.117480 10633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:34.117897 10633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:34.118053 10633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:34.129417 10633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:34.129523 10633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:34.129567 10633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:34.129623 10633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:34.129666 10633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:34.139674 10633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37357,127.3.240.253:34805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.253.keytab
--principal=oryx/127.3.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:34805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
W20260927 23:04:34.139994 10633 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.240.254:37357,127.3.240.253:34805'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260927 23:04:34.140956 10633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:34.142171 10633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:34.153486 10638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:34.153533 10639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:34.153569 10641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:34.154470 10633 server_base.cc:1034] running on GCE node
I20260927 23:04:34.155220 10633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:34.156461 10633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:34.157691 10633 hybrid_clock.cc:648] HybridClock initialized: now 1790550274157625 us; error 39 us; skew 500 ppm
I20260927 23:04:34.158020 10633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:34.161195 10633 init.cc:377] Logged in from keytab as oryx/127.3.240.253@KRBTEST.COM (short username oryx)
I20260927 23:04:34.162482 10633 webserver.cc:460] Webserver started at http://127.3.240.253:37707/ using document root <none> and password file <none>
I20260927 23:04:34.163200 10633 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:34.163367 10633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:34.163645 10633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:34.166286 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data/instance:
uuid: "e15b5882405a4f9e97888ddd00fae3d6"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "732b998687eb6eacb4f1e8acf9fa9b0b"
server_key_iv: "c6206476b16b23034f82660a4a17499b"
server_key_version: "encryptionkey@0"
I20260927 23:04:34.166975 10633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal/instance:
uuid: "e15b5882405a4f9e97888ddd00fae3d6"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "732b998687eb6eacb4f1e8acf9fa9b0b"
server_key_iv: "c6206476b16b23034f82660a4a17499b"
server_key_version: "encryptionkey@0"
I20260927 23:04:34.171988 10633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 23:04:34.175179 10648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:34.176432 10633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:04:34.176602 10633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
uuid: "e15b5882405a4f9e97888ddd00fae3d6"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "732b998687eb6eacb4f1e8acf9fa9b0b"
server_key_iv: "c6206476b16b23034f82660a4a17499b"
server_key_version: "encryptionkey@0"
I20260927 23:04:34.176796 10633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:34.214588 10633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:34.218291 10633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:34.218534 10633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:34.236429 10633 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:34805
I20260927 23:04:34.236445 10699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:34805 every 8 connection(s)
I20260927 23:04:34.237730 10633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
I20260927 23:04:34.241271 10700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:34.241869 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 10633
I20260927 23:04:34.249985 10700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.253@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:34.263159 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.244060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.3.240.253:34805)
0927 23:04:34.244803 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.244815 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.244853 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:04:34.246296 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.246315 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.246660 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.247071 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.248431 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.249600 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.250856 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.251515 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.251685 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.255829 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.255839 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.255850 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.255903 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.259150 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.259843 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.259850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.259864 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.260044 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.260507 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.260512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.260515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.261053 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.261345 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.261729 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.262085 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":306,"threads_started":1}
I20260927 23:04:34.268507 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.252091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.3.240.254:37357)
0927 23:04:34.254536 (+ 2445us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.254541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.254554 (+ 13us) server_negotiation.cc:407] Connection header received
0927 23:04:34.254595 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.254616 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.254667 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.254783 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.255896 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.256928 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.258874 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.259482 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.259589 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.263115 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.263120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.263123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.263151 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.265928 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.266570 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.266579 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.266582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.266699 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.267117 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.267122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.267126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.267400 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.267554 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.267971 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.268154 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2356,"thread_start_us":2255,"threads_started":1}
I20260927 23:04:34.268541 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.252073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:45092)
0927 23:04:34.252818 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.252864 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.253021 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.254852 (+ 1831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.254858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.255455 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.255707 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.255718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.257097 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.257107 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.258717 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.258732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.259790 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.259796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.260002 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.260632 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.260652 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.262911 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.266057 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.266066 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.266073 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.266444 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.266804 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.266810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.266813 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.266948 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.267527 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.267534 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.267692 (+ 158us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.267874 (+ 182us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.268154 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":392,"threads_started":1}
I20260927 23:04:34.269989 10700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.253@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260927 23:04:34.280303 10705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.270521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.0.0.1:41566)
0927 23:04:34.270696 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.270713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.270806 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.271161 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.271164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.271428 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.271549 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.271556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.272442 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.272446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.273312 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.273322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.273917 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.273921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.274040 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.274483 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.274497 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.276184 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.278464 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.278467 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.278469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.278772 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.279142 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.279145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.279147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.279279 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.279677 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.279679 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.279809 (+ 130us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.279915 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.280121 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 23:04:34.280553 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.270732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41566 (local address 127.3.240.253:34805)
0927 23:04:34.270867 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.270871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.270882 (+ 11us) server_negotiation.cc:407] Connection header received
0927 23:04:34.270913 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.270916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.270964 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.271109 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.271650 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.272311 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.273466 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.273807 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.273893 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.276322 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.276326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.276329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.276353 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.278353 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.278861 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.278864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.278866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.278984 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.279358 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.279361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.279363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.279597 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.279687 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.280008 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.280251 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":56}
I20260927 23:04:34.284703 10700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Bootstrap starting.
I20260927 23:04:34.287587 10700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:34.288038 10495 heartbeater.cc:503] Master 127.3.240.254:37357 was elected leader, sending a full tablet report...
I20260927 23:04:34.288727 10700 log.cc:824] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:34.291045 10700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: No bootstrap required, opened a new log
I20260927 23:04:34.297488 10700 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } }
I20260927 23:04:34.297765 10700 raft_consensus.cc:374] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:34.297822 10700 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e15b5882405a4f9e97888ddd00fae3d6, State: Initialized, Role: FOLLOWER
I20260927 23:04:34.298337 10700 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } }
I20260927 23:04:34.299124 10707 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } } }
I20260927 23:04:34.299438 10707 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:04:34.299978 10700 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 23:04:34.307790 10718 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:04:34.307929 10718 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:04:34.309286 10633 master_runner.cc:428] Detected that this master e15b5882405a4f9e97888ddd00fae3d6 is joining an existing cluster
I20260927 23:04:34.309376 10633 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.240.253:34805
I20260927 23:04:34.328187 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.313109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45094 (local address 127.3.240.254:37357)
0927 23:04:34.313293 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.313301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.313799 (+ 498us) server_negotiation.cc:407] Connection header received
0927 23:04:34.313998 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.314002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.314081 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.314174 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.315291 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.316246 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.318225 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.318917 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.319038 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.321662 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.321670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.321676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.321713 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.325130 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.325896 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.325902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.325906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.326041 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.326500 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.326505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.326509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.326851 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.327022 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.327597 (+ 575us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.327823 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:04:34.329169 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.313179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:45094)
0927 23:04:34.313712 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.313728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.313890 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.314191 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.314195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.314726 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.315052 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.315076 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.316404 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.316408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.318059 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.318086 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.319349 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.319358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.319554 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.320238 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.320271 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.321462 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.325255 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.325262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.325267 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.325770 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.326173 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.326180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.326184 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.326360 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.327084 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.327089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.327300 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.328362 (+ 1062us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.328679 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":516,"threads_started":1}
I20260927 23:04:34.329628 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.313226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.0.0.1:41580)
0927 23:04:34.315462 (+ 2236us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.315477 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.315590 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.315973 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.315977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.316590 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.316925 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.316934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.318345 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.318349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.319814 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.319829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.321197 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.321206 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.321397 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.321987 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.322011 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.323168 (+ 1157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.326755 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.326762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.326765 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.327319 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.327712 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.327717 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.327722 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.327896 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.328572 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.328579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.328727 (+ 148us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.328878 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.329191 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2134,"thread_start_us":1937,"threads_started":1}
I20260927 23:04:34.330506 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.313499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.3.240.253:34805)
0927 23:04:34.313687 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.313692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.315565 (+ 1873us) server_negotiation.cc:407] Connection header received
0927 23:04:34.315681 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.315685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.315745 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.315832 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.317055 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.318152 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.319961 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.320777 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.320905 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.323337 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.323345 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.323352 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.323387 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.326652 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.327447 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.327452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.327455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.327588 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.328022 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.328028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.328034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.328372 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.328515 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.330067 (+ 1552us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.330250 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
WARNING: no policy specified for oryx/127.3.240.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.252@KRBTEST.COM" created.
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550272, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260927 23:04:34.355691 10703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.335698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.3.240.253:34805)
0927 23:04:34.335877 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.335884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.339744 (+ 3860us) server_negotiation.cc:407] Connection header received
0927 23:04:34.340080 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.340087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.340178 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.340273 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.341675 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.342952 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.344268 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.345027 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.345151 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.349235 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.349243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.349248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.349281 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.352771 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.353592 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.353597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.353601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.353743 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.354152 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.354159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.354162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.354480 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.354622 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.355172 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.355370 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 23:04:34.355897 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.335654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.0.0.1:41588)
0927 23:04:34.339593 (+ 3939us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.339662 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.339871 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.340343 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.340350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.341191 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.341506 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.341518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.343094 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.343098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.344141 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.344152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.345192 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.345200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.345419 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.346096 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.346123 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.349060 (+ 2937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.352893 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.352905 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.352915 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.353472 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.353867 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.353875 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.353881 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.354041 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.354629 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.354640 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.354853 (+ 213us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.355074 (+ 221us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.355388 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3647,"thread_start_us":3484,"threads_started":1}
I20260927 23:04:34.358021 10172 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.240.253:34805: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6"
member_type: NON_VOTER
last_known_addr {
host: "127.3.240.253"
port: 34805
}
attrs {
promote: true
}
}
dest_uuid: "a56b7432116b47e8ac17984a7421f31d"
cas_config_opid_index: -1
I20260927 23:04:34.359220 10172 consensus_queue.cc:237] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } }
I20260927 23:04:34.366171 10731 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e15b5882405a4f9e97888ddd00fae3d6 (127.3.240.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } }
I20260927 23:04:34.367170 10675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:34.370919 10731 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.240.253:34805
I20260927 23:04:34.371124 10675 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Refusing update from remote peer a56b7432116b47e8ac17984a7421f31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260927 23:04:34.371616 10633 master.cc:537] Master@127.3.240.253:34805 shutting down...
I20260927 23:04:34.371697 10730 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } }
I20260927 23:04:34.372059 10730 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:34.372442 10731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 23:04:34.376016 10158 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260927 23:04:34.377678 10158 consensus_peers.cc:487] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d -> Peer e15b5882405a4f9e97888ddd00fae3d6 (127.3.240.253:34805): Couldn't send request to peer e15b5882405a4f9e97888ddd00fae3d6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.252.keytab.
Entry for principal oryx/127.3.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.252.keytab.
I20260927 23:04:34.382532 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:35827
--webserver_interface=127.3.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.252.keytab
--principal=oryx/127.3.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37357,127.3.240.253:34805,127.3.240.252:35827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:04:34.386247 10633 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:34.386642 10633 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:34.386811 10633 tablet_replica.cc:330] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: stopping tablet replica
I20260927 23:04:34.392882 10633 master.cc:552] Master@127.3.240.253:34805 shutdown complete.
I20260927 23:04:34.392983 10633 master_runner.cc:305] Clearing existing system tablet
I20260927 23:04:34.393363 10633 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:34.393467 10633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:34.395514 10633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.003s
I20260927 23:04:34.396903 10736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:34.397224 10633 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:34.397478 10633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260927 23:04:34.397560 10633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
uuid: "e15b5882405a4f9e97888ddd00fae3d6"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "732b998687eb6eacb4f1e8acf9fa9b0b"
server_key_iv: "c6206476b16b23034f82660a4a17499b"
server_key_version: "encryptionkey@0"
I20260927 23:04:34.401072 10633 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:34.405859 10633 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 23:04:34.406039 10633 log.cc:1196] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260927 23:04:34.406438 10633 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Deleting consensus metadata
I20260927 23:04:34.406710 10633 master_runner.cc:315] Copying system tablet from 127.3.240.254:37357
I20260927 23:04:34.414841 10633 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: tablet copy: Beginning tablet copy session from remote peer at address 127.3.240.254:37357
I20260927 23:04:34.430445 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.415605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.3.240.254:37357)
0927 23:04:34.415812 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.415821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.419526 (+ 3705us) server_negotiation.cc:407] Connection header received
0927 23:04:34.419741 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.419746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.419833 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.419949 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.420781 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.421563 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.422823 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.423340 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.423480 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.425273 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.425279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.425285 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.425322 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.427903 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.428573 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.428578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.428582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.428716 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.429076 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.429083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.429087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.429347 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.429502 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.429867 (+ 365us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.430076 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:04:34.430518 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.415568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:45108)
0927 23:04:34.419383 (+ 3815us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.419401 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.419589 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.419969 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.419973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.420354 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.420637 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.420648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.421723 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.421729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.422674 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.422686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.423493 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.423499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.423700 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.424366 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.424383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.425065 (+ 682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.428042 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.428049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.428053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.428453 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.428826 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.428831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.428835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.428968 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.429469 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.429474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.429631 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.429775 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.430064 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3697,"thread_start_us":3594,"threads_started":1}
I20260927 23:04:34.431550 10192 tablet_copy_service.cc:140] P a56b7432116b47e8ac17984a7421f31d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e15b5882405a4f9e97888ddd00fae3d6 ({username='oryx', principal='oryx/127.3.240.253@KRBTEST.COM'} at 127.0.0.1:45108)
I20260927 23:04:34.431823 10192 tablet_copy_service.cc:161] P a56b7432116b47e8ac17984a7421f31d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e15b5882405a4f9e97888ddd00fae3d6 at {username='oryx', principal='oryx/127.3.240.253@KRBTEST.COM'} at 127.0.0.1:45108: session id = e15b5882405a4f9e97888ddd00fae3d6-00000000000000000000000000000000
I20260927 23:04:34.434911 10192 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:04:34.436695 10633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 23:04:34.441337 10633 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: tablet copy: Starting download of 0 data blocks...
I20260927 23:04:34.441613 10633 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: tablet copy: Starting download of 1 WAL segments...
I20260927 23:04:34.443881 10633 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:04:34.447048 10192 tablet_copy_service.cc:342] P a56b7432116b47e8ac17984a7421f31d: Request end of tablet copy session e15b5882405a4f9e97888ddd00fae3d6-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.240.253@KRBTEST.COM'} at 127.0.0.1:45108
I20260927 23:04:34.447234 10192 tablet_copy_service.cc:433] P a56b7432116b47e8ac17984a7421f31d: ending tablet copy session e15b5882405a4f9e97888ddd00fae3d6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e15b5882405a4f9e97888ddd00fae3d6
I20260927 23:04:34.457597 10633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:34.460525 10743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:34.470136 10744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:34.477013 10746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:34.477401 10633 server_base.cc:1034] running on GCE node
I20260927 23:04:34.477751 10633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:34.478408 10633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:34.479637 10633 hybrid_clock.cc:648] HybridClock initialized: now 1790550274479557 us; error 47 us; skew 500 ppm
I20260927 23:04:34.479841 10633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:34.481971 10633 init.cc:377] Logged in from keytab as oryx/127.3.240.253@KRBTEST.COM (short username oryx)
I20260927 23:04:34.483866 10633 webserver.cc:460] Webserver started at http://127.3.240.253:44583/ using document root <none> and password file <none>
I20260927 23:04:34.484225 10633 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:34.484298 10633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:34.487190 10633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:34.489053 10753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:34.489414 10633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260927 23:04:34.489548 10633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
uuid: "e15b5882405a4f9e97888ddd00fae3d6"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "732b998687eb6eacb4f1e8acf9fa9b0b"
server_key_iv: "c6206476b16b23034f82660a4a17499b"
server_key_version: "encryptionkey@0"
I20260927 23:04:34.489698 10633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:34.524775 10633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:34.528671 10633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:34.551870 10633 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:34805
I20260927 23:04:34.551908 10804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:34805 every 8 connection(s)
I20260927 23:04:34.552837 10633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
I20260927 23:04:34.557857 10805 sys_catalog.cc:263] Verifying existing consensus state
I20260927 23:04:34.561980 10805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Bootstrap starting.
I20260927 23:04:34.577569 10805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:04:34.578156 10805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: Bootstrap complete.
I20260927 23:04:34.579214 10805 raft_consensus.cc:348] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } }
I20260927 23:04:34.579579 10805 raft_consensus.cc:729] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e15b5882405a4f9e97888ddd00fae3d6, State: Initialized, Role: LEARNER
I20260927 23:04:34.579841 10805 consensus_queue.cc:260] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } }
I20260927 23:04:34.580785 10808 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } }
I20260927 23:04:34.581053 10808 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: This master's current role is: LEARNER
I20260927 23:04:34.581156 10805 sys_catalog.cc:564] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:34.589967 10819 catalog_manager.cc:1259] Loaded cluster ID: 5d16037075ed46b783798781536f95e9
I20260927 23:04:34.590066 10819 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: loading cluster ID for follower catalog manager: success
I20260927 23:04:34.594061 10819 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: acquiring CA information for follower catalog manager: success
I20260927 23:04:34.596758 10819 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 23:04:34.755198 10628 heartbeater.cc:503] Master 127.3.240.254:37357 was elected leader, sending a full tablet report...
W20260927 23:04:34.761262 10734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:34.761519 10734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:34.761605 10734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:34.769002 10734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:34.769090 10734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:34.769132 10734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:34.769183 10734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:34.769224 10734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:34.779428 10734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37357,127.3.240.253:34805,127.3.240.252:35827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.252.keytab
--principal=oryx/127.3.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:35827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:34.781049 10734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:34.783079 10734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 23:04:34.785746 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.778474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.0.0.1:41602)
0927 23:04:34.778660 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.778679 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.778807 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.779540 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.779546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.779575 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:34.779776 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.779788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.781336 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.781344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.783841 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.783857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.784997 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.785003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.785219 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.785269 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.785365 (+ 96us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.785506 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 23:04:34.785784 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.778657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.3.240.253:34805)
0927 23:04:34.779224 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.779228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.779246 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:04:34.779304 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.779308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.779370 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.779474 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:34.779925 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.781195 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.784009 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.784843 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.784939 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.785094 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.785433 (+ 339us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.785543 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":349,"threads_started":1}
I20260927 23:04:34.786953 10780 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260927 23:04:34.787569 10808 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a56b7432116b47e8ac17984a7421f31d. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } }
I20260927 23:04:34.787819 10808 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: This master's current role is: LEARNER
W20260927 23:04:34.795930 10825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:34.795893 10828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:34.796056 10826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:34.796964 10734 server_base.cc:1034] running on GCE node
I20260927 23:04:34.797698 10734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:34.798893 10734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:34.800086 10734 hybrid_clock.cc:648] HybridClock initialized: now 1790550274800040 us; error 44 us; skew 500 ppm
I20260927 23:04:34.800356 10734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:34.803340 10734 init.cc:377] Logged in from keytab as oryx/127.3.240.252@KRBTEST.COM (short username oryx)
I20260927 23:04:34.804631 10734 webserver.cc:460] Webserver started at http://127.3.240.252:38501/ using document root <none> and password file <none>
I20260927 23:04:34.805205 10734 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:34.805294 10734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:34.805523 10734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:34.808236 10734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data/instance:
uuid: "c5880468b1f946c9a4e3015789415f1d"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "754790c0832ac2a7636b480b2c84295d"
server_key_iv: "7345f3f1ddbf41cbb79fdaf3df0e408d"
server_key_version: "encryptionkey@0"
I20260927 23:04:34.808856 10734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal/instance:
uuid: "c5880468b1f946c9a4e3015789415f1d"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "754790c0832ac2a7636b480b2c84295d"
server_key_iv: "7345f3f1ddbf41cbb79fdaf3df0e408d"
server_key_version: "encryptionkey@0"
I20260927 23:04:34.813468 10734 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 23:04:34.817076 10835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:34.818225 10734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 23:04:34.818384 10734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
uuid: "c5880468b1f946c9a4e3015789415f1d"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "754790c0832ac2a7636b480b2c84295d"
server_key_iv: "7345f3f1ddbf41cbb79fdaf3df0e408d"
server_key_version: "encryptionkey@0"
I20260927 23:04:34.818570 10734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:34.834502 10734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:34.838295 10734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:34.838552 10734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:34.855350 10734 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:35827
I20260927 23:04:34.855379 10886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:35827 every 8 connection(s)
I20260927 23:04:34.856446 10734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
I20260927 23:04:34.859476 10887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:34.865222 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 10734
I20260927 23:04:34.865628 10887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260927 23:04:34.885406 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.868359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.3.240.254:37357)
0927 23:04:34.868535 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.868540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.868925 (+ 385us) server_negotiation.cc:407] Connection header received
0927 23:04:34.869916 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.869924 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.870008 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.870112 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.871601 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.872744 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.874575 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.875228 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.875352 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.879350 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.879354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.879357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.879387 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.882406 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.883223 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.883228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.883232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.883376 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.883815 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.883819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.883822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.884121 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.884262 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.884879 (+ 617us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.885089 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 23:04:34.886409 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.867163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45128 (local address 127.3.240.252:35827)
0927 23:04:34.867809 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.867818 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.867849 (+ 31us) server_negotiation.cc:407] Connection header received
0927 23:04:34.869676 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.869695 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.869983 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.870148 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.871803 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.873279 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.874858 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.875642 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.875745 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.880545 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.880551 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.880561 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.880585 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.883653 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.884351 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.884356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.884358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.884508 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.884839 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.884843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.884845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.885136 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.885436 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.885673 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.885887 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"spinlock_wait_cycles":11520,"thread_start_us":300,"threads_started":1}
I20260927 23:04:34.886440 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.868130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:45120)
0927 23:04:34.868787 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.868822 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.869688 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.870180 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.870185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.870899 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.871447 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.871469 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.872902 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.872909 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.874410 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.874422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.875424 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.875429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.875621 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.876279 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.876318 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.879190 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.882518 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.882525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.882532 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.883117 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.883497 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.883500 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.883502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.883695 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.884242 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.884249 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.884545 (+ 296us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.884791 (+ 246us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.885220 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":286,"threads_started":1}
I20260927 23:04:34.888423 10887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260927 23:04:34.899475 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.889094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.0.0.1:41616)
0927 23:04:34.889258 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.889274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.889389 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.889706 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.889710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.890013 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.890115 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.890121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.891135 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.891140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.891987 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.891996 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.892639 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.892643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.892766 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.893260 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.893275 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.895342 (+ 2067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.897688 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.897691 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.897694 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.898016 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.898304 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.898307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.898309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.898417 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.898852 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.898854 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.898961 (+ 107us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.899115 (+ 154us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.899283 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 23:04:34.899530 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.889089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41616 (local address 127.3.240.253:34805)
0927 23:04:34.889257 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.889263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.889340 (+ 77us) server_negotiation.cc:407] Connection header received
0927 23:04:34.889512 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.889517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.889582 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.889673 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.890257 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.890980 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.892114 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.892523 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.892641 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.895495 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.895500 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.895502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.895528 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.897583 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.898104 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.898107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.898109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.898218 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.898508 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.898511 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.898513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.898765 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.898863 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.899178 (+ 315us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.899328 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:04:34.900416 10887 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550274, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260927 23:04:34.911237 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.900923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35827 (local address 127.0.0.1:45130)
0927 23:04:34.901094 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.901108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.901200 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.901482 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.901486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.901798 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.901904 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.901911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.902768 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.902776 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.903580 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.903590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.904161 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.904166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.904346 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.904726 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.904739 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.906520 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.909029 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.909035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.909038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.909365 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.909698 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.909705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.909709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.909846 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.910285 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.910289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.910430 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.910559 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.910807 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 23:04:34.911265 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.901142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.3.240.252:35827)
0927 23:04:34.901288 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.901293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.901305 (+ 12us) server_negotiation.cc:407] Connection header received
0927 23:04:34.901341 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.901343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.901382 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.901470 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.902026 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.902658 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.903699 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.904042 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.904158 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.906682 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.906686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.906690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.906713 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.908908 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.909472 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.909480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.909498 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.909598 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.909944 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.909951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.909954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.910181 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.910316 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.910625 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.910918 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":46}
I20260927 23:04:34.916018 10887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Bootstrap starting.
I20260927 23:04:34.918746 10887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:34.919715 10887 log.cc:824] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:34.922361 10887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: No bootstrap required, opened a new log
I20260927 23:04:34.928942 10887 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } }
I20260927 23:04:34.929205 10887 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:34.929280 10887 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5880468b1f946c9a4e3015789415f1d, State: Initialized, Role: FOLLOWER
I20260927 23:04:34.929766 10887 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } }
I20260927 23:04:34.930536 10893 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } } }
I20260927 23:04:34.931275 10893 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:04:34.931512 10887 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: configured and running, proceeding with master startup.
W20260927 23:04:34.940066 10904 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:04:34.940207 10904 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:04:34.941424 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 10233
I20260927 23:04:34.944950 10734 master_runner.cc:428] Detected that this master c5880468b1f946c9a4e3015789415f1d is joining an existing cluster
I20260927 23:04:34.945077 10734 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.240.252:35827
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.193@KRBTEST.COM
I20260927 23:04:34.963724 10906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.942451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.3.240.193:36617)
0927 23:04:34.943062 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.943071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.943101 (+ 30us) server_negotiation.cc:407] Connection header received
0927 23:04:34.943162 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.943169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.943396 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.943612 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.945204 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.946758 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.948620 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.949447 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.950122 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.955835 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.955846 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.955861 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.955897 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.959729 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.960592 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.960600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.960603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.960735 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.961221 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.961228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.961233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.961615 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.962898 (+ 1283us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.963071 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.963324 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":290,"threads_started":1}
I20260927 23:04:34.965471 10356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:34.972594 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.949170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.3.240.254:37357)
0927 23:04:34.949329 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.949336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.950042 (+ 706us) server_negotiation.cc:407] Connection header received
0927 23:04:34.954283 (+ 4241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.954291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.954399 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.954480 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.956663 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.958083 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.960671 (+ 2588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.961499 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.961612 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.964988 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.964996 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.965002 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.965049 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.968385 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.969113 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.969120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.969124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.969258 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.969687 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.969692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.969696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.970051 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.970158 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.971760 (+ 1602us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.972061 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:04:34.973686 10911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.949173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:45124)
0927 23:04:34.949928 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.949959 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.954100 (+ 4141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.954713 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.954720 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.955529 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.956454 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.956471 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.958296 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.958303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.960477 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.960495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.961945 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.961953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.962195 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.962955 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.962985 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.964341 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.968526 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.968531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.968536 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.968985 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.969387 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.969391 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.969395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.969568 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.970223 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.970227 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.970401 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.970566 (+ 165us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.970833 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":470,"threads_started":1}
I20260927 23:04:34.982070 10823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.949463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41630 (local address 127.3.240.253:34805)
0927 23:04:34.949632 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.949639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.965345 (+ 15706us) server_negotiation.cc:407] Connection header received
0927 23:04:34.965581 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.965588 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.965666 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.965732 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.967060 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.968124 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.970001 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.970564 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.970640 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.974316 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.974327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.974331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.974387 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.979105 (+ 4718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.979864 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.979869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.979872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.980002 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.980462 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.980467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.980471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.980780 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.980903 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.981544 (+ 641us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.981752 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:04:34.983183 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.949797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.0.0.1:41630)
0927 23:04:34.965256 (+ 15459us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.965272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.965439 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.965951 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.965957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.966558 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.966869 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.966886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.968323 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.968329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.969849 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.969861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.970876 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.970882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.971128 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.972291 (+ 1163us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.972316 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.974150 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.979247 (+ 5097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.979252 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.979256 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.979725 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.980138 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.980143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.980147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.980329 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.981007 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.981012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.981240 (+ 228us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.982330 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.982675 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15325,"thread_start_us":15464,"threads_started":1}
Sep 27 23:04:34 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550272, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260927 23:04:34.996376 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.979202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35827 (local address 127.0.0.1:45144)
0927 23:04:34.979336 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.979354 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.979452 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.980408 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.980413 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.980833 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.981014 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.981024 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.983078 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.983084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.984419 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.984432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.985766 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.985772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.985923 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.986461 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.986482 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.989362 (+ 2880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.993686 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.993691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.993695 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.994180 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.994622 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.994628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.994631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.994777 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.995466 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.995471 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.995625 (+ 154us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.995758 (+ 133us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.996019 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260927 23:04:34.997305 10890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.965752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.3.240.252:35827)
0927 23:04:34.965915 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.965923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.972427 (+ 6504us) server_negotiation.cc:407] Connection header received
0927 23:04:34.972622 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.972627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.972695 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.972759 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.975058 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.976085 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.986404 (+ 10319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.987151 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.987244 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.990672 (+ 3428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.990677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.990681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.990708 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.994633 (+ 3925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.995402 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.995407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.995411 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.995534 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.995880 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.995885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.995888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.996200 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.996301 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.996776 (+ 475us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.996978 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260927 23:04:34.997944 10915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.979467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.3.240.252:35827)
0927 23:04:34.980048 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:34.980054 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:34.980073 (+ 19us) server_negotiation.cc:407] Connection header received
0927 23:04:34.980152 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:34.980156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:34.980224 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:34.981140 (+ 916us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:34.981443 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.982888 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.984797 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.985607 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.989511 (+ 3904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.989739 (+ 228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:34.989746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:34.989750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:34.989779 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:34.993523 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.994341 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.994345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.994349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.994494 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:34.994917 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:34.994922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:34.994925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:34.995331 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:34.996964 (+ 1633us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:34.997092 (+ 128us) server_negotiation.cc:299] Negotiation successful
0927 23:04:34.997284 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":434,"thread_start_us":283,"threads_started":1}
I20260927 23:04:34.998237 10914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:34.965740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35827 (local address 127.0.0.1:45132)
0927 23:04:34.972347 (+ 6607us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:34.972364 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:34.972510 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:34.972930 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:34.972942 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:34.973706 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:34.973992 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.974006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.984555 (+ 10549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.984561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.986254 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:34.986269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:34.987822 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:34.987829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:34.988029 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:34.988575 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:34.988595 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:34.990514 (+ 1919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:34.994765 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.994770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.994774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.995302 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.995649 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:34.995654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:34.995657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:34.995785 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:34.996366 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:34.996370 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:34.996511 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 23:04:34.997465 (+ 954us) client_negotiation.cc:240] Negotiation successful
0927 23:04:34.997727 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6480,"mutex_wait_us":1,"thread_start_us":6269,"threads_started":1}
I20260927 23:04:34.999591 10172 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.240.252:35827: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c5880468b1f946c9a4e3015789415f1d"
member_type: NON_VOTER
last_known_addr {
host: "127.3.240.252"
port: 35827
}
attrs {
promote: true
}
}
dest_uuid: "a56b7432116b47e8ac17984a7421f31d"
cas_config_opid_index: 5
I20260927 23:04:35.000564 10172 consensus_queue.cc:237] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } }
I20260927 23:04:35.002345 10731 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c5880468b1f946c9a4e3015789415f1d (127.3.240.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.002912 10780 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 LEARNER]: Refusing update from remote peer a56b7432116b47e8ac17984a7421f31d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 23:04:35.003331 10862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:35.003512 10916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260927 23:04:35.005352 10731 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.240.252:35827
I20260927 23:04:35.005650 10731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.005888 10731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:35.015472 10780 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c5880468b1f946c9a4e3015789415f1d (127.3.240.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.019541 10808 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: true } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.019801 10808 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: This master's current role is: LEARNER
I20260927 23:04:35.020085 10917 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: attempting to promote NON_VOTER e15b5882405a4f9e97888ddd00fae3d6 to VOTER
I20260927 23:04:35.020799 10917 consensus_queue.cc:237] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } }
I20260927 23:04:35.023033 10779 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 LEARNER]: Refusing update from remote peer a56b7432116b47e8ac17984a7421f31d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 23:04:35.023661 10917 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260927 23:04:35.025714 10916 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e15b5882405a4f9e97888ddd00fae3d6 (127.3.240.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.027287 10734 master.cc:537] Master@127.3.240.252:35827 shutting down...
I20260927 23:04:35.028216 10862 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Refusing update from remote peer a56b7432116b47e8ac17984a7421f31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260927 23:04:35.029310 10861 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Refusing update from remote peer a56b7432116b47e8ac17984a7421f31d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260927 23:04:35.029789 10916 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.030071 10916 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:35.030300 10916 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.030519 10916 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:35.031951 10779 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e15b5882405a4f9e97888ddd00fae3d6 (127.3.240.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.033981 10731 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:35.037910 10808 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.038250 10808 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:04:35.040261 10734 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:35.040617 10734 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:35.040802 10734 tablet_replica.cc:330] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: stopping tablet replica
I20260927 23:04:35.047986 10734 master.cc:552] Master@127.3.240.252:35827 shutdown complete.
I20260927 23:04:35.048064 10734 master_runner.cc:305] Clearing existing system tablet
I20260927 23:04:35.048425 10734 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:35.048537 10734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260927 23:04:35.049183 10158 consensus_peers.cc:487] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d -> Peer c5880468b1f946c9a4e3015789415f1d (127.3.240.252:35827): Couldn't send request to peer c5880468b1f946c9a4e3015789415f1d. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260927 23:04:35.051422 10734 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260927 23:04:35.052955 10922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:35.053298 10734 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:35.053536 10734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 23:04:35.053628 10734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
uuid: "c5880468b1f946c9a4e3015789415f1d"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "754790c0832ac2a7636b480b2c84295d"
server_key_iv: "7345f3f1ddbf41cbb79fdaf3df0e408d"
server_key_version: "encryptionkey@0"
I20260927 23:04:35.058380 10734 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:35.064853 10734 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260927 23:04:35.065002 10734 log.cc:1196] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260927 23:04:35.065410 10734 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Deleting consensus metadata
I20260927 23:04:35.065639 10734 master_runner.cc:315] Copying system tablet from 127.3.240.254:37357
I20260927 23:04:35.067945 10734 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: tablet copy: Beginning tablet copy session from remote peer at address 127.3.240.254:37357
I20260927 23:04:35.082686 10704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.068690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.3.240.254:37357)
0927 23:04:35.068915 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:35.068921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:35.069345 (+ 424us) server_negotiation.cc:407] Connection header received
0927 23:04:35.069590 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:35.069596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:35.069672 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:35.069773 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:35.070731 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.071803 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.073605 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.074141 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.074251 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.076298 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:35.076304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:35.076308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:35.076343 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:35.079686 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.080418 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.080423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.080427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.080587 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.081273 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.081277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.081280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.081579 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:35.081721 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:35.082148 (+ 427us) server_negotiation.cc:299] Negotiation successful
0927 23:04:35.082354 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 23:04:35.082917 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.068640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:45140)
0927 23:04:35.069246 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:35.069264 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:35.069446 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:35.069827 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:35.069832 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:35.070320 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:35.070561 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.070579 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.071989 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.071995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.073472 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.073485 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.074290 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.074296 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.074464 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.075129 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:35.075150 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:35.076099 (+ 949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:35.079826 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:35.079831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:35.079835 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:35.080294 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:35.080700 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:35.080705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:35.080709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:35.080867 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:35.081696 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:35.081701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:35.081877 (+ 176us) client_negotiation.cc:769] Sending connection context
0927 23:04:35.082081 (+ 204us) client_negotiation.cc:240] Negotiation successful
0927 23:04:35.082406 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":384,"threads_started":1}
I20260927 23:04:35.083633 10192 tablet_copy_service.cc:140] P a56b7432116b47e8ac17984a7421f31d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c5880468b1f946c9a4e3015789415f1d ({username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:45140)
I20260927 23:04:35.083797 10192 tablet_copy_service.cc:161] P a56b7432116b47e8ac17984a7421f31d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c5880468b1f946c9a4e3015789415f1d at {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:45140: session id = c5880468b1f946c9a4e3015789415f1d-00000000000000000000000000000000
I20260927 23:04:35.087383 10192 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: Tablet Copy: opened 0 blocks and 1 log segments
I20260927 23:04:35.089062 10734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260927 23:04:35.095481 10734 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: tablet copy: Starting download of 0 data blocks...
I20260927 23:04:35.095763 10734 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: tablet copy: Starting download of 1 WAL segments...
I20260927 23:04:35.098515 10734 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260927 23:04:35.102530 10192 tablet_copy_service.cc:342] P a56b7432116b47e8ac17984a7421f31d: Request end of tablet copy session c5880468b1f946c9a4e3015789415f1d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:45140
I20260927 23:04:35.102739 10192 tablet_copy_service.cc:433] P a56b7432116b47e8ac17984a7421f31d: ending tablet copy session c5880468b1f946c9a4e3015789415f1d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c5880468b1f946c9a4e3015789415f1d
I20260927 23:04:35.107573 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 10233
I20260927 23:04:35.129575 10734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:35.132090 10929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:35.135885 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:36617
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:37357
--webserver_port=41267
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805,127.3.240.252:35827 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:35.135984 10930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:35.143888 10734 server_base.cc:1034] running on GCE node
W20260927 23:04:35.144193 10932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:35.144652 10734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:35.145350 10734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:35.146519 10734 hybrid_clock.cc:648] HybridClock initialized: now 1790550275146477 us; error 36 us; skew 500 ppm
I20260927 23:04:35.146726 10734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:35 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550275, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:35.148504 10734 init.cc:377] Logged in from keytab as oryx/127.3.240.252@KRBTEST.COM (short username oryx)
I20260927 23:04:35.155912 10734 webserver.cc:460] Webserver started at http://127.3.240.252:46107/ using document root <none> and password file <none>
I20260927 23:04:35.156229 10734 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:35.156339 10734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:35.158625 10734 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260927 23:04:35.160259 10940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:35.160615 10734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260927 23:04:35.160815 10734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
uuid: "c5880468b1f946c9a4e3015789415f1d"
format_stamp: "Formatted at 2026-09-27 23:04:34 on dist-test-slave-lmfp"
server_key: "754790c0832ac2a7636b480b2c84295d"
server_key_iv: "7345f3f1ddbf41cbb79fdaf3df0e408d"
server_key_version: "encryptionkey@0"
I20260927 23:04:35.161011 10734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:35.168056 10734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:35.171247 10734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:35.253567 10734 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:35827
I20260927 23:04:35.253592 10991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:35827 every 8 connection(s)
I20260927 23:04:35.254520 10734 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
I20260927 23:04:35.267601 10992 sys_catalog.cc:263] Verifying existing consensus state
I20260927 23:04:35.272281 10992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Bootstrap starting.
I20260927 23:04:35.285800 10992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:04:35.286206 10992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Bootstrap complete.
I20260927 23:04:35.287032 10992 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } }
I20260927 23:04:35.287277 10992 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c5880468b1f946c9a4e3015789415f1d, State: Initialized, Role: LEARNER
I20260927 23:04:35.287509 10992 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } }
I20260927 23:04:35.288255 10995 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.288584 10992 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:35.288740 10995 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: This master's current role is: LEARNER
I20260927 23:04:35.299968 11006 catalog_manager.cc:1259] Loaded cluster ID: 5d16037075ed46b783798781536f95e9
I20260927 23:04:35.300071 11006 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: loading cluster ID for follower catalog manager: success
I20260927 23:04:35.302749 11006 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: acquiring CA information for follower catalog manager: success
I20260927 23:04:35.304387 11006 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 23:04:35.535574 10934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:35.535871 10934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:35.535971 10934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:35.543270 10934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:35.543372 10934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:35.543493 10934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:35.552624 10934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:36617
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=41267
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805,127.3.240.252:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:35.553819 10934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:35.555230 10934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:35.570092 11014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:35.570209 11012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:35.570531 11011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:35.571050 10934 server_base.cc:1034] running on GCE node
I20260927 23:04:35.571990 10934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:35.573091 10934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:35.574335 10934 hybrid_clock.cc:648] HybridClock initialized: now 1790550275574290 us; error 50 us; skew 500 ppm
I20260927 23:04:35.574662 10934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:35.574910 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.566588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.3.240.252:35827)
0927 23:04:35.567236 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:35.567241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:35.567266 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:04:35.567339 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:35.567345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:35.567478 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:35.567607 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:35.568245 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.569797 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.572650 (+ 2853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.573927 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.574071 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.574294 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:35.574491 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 23:04:35.574616 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":303,"threads_started":1}
I20260927 23:04:35.574910 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.566324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35827 (local address 127.0.0.1:45154)
0927 23:04:35.566839 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:35.566865 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:35.567062 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:35.567835 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:35.567840 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:35.567873 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:35.568082 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.568092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.569976 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.569982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.572462 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.572477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.574131 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.574137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.574302 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.574340 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 23:04:35.574466 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 23:04:35.574596 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":237,"threads_started":1}
I20260927 23:04:35.576112 10967 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260927 23:04:35.576611 10995 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a56b7432116b47e8ac17984a7421f31d. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:35.576898 10995 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: This master's current role is: LEARNER
Sep 27 23:04:35 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550275, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:35.577977 10934 init.cc:377] Logged in from keytab as oryx/127.3.240.193@KRBTEST.COM (short username oryx)
I20260927 23:04:35.579344 10934 webserver.cc:460] Webserver started at http://127.3.240.193:41267/ using document root <none> and password file <none>
I20260927 23:04:35.579955 10934 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:35.580102 10934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:35.585115 10934 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 23:04:35.587971 11024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:35.589224 10934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:35.589382 10934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "e4fd87ed1cbf47b39818dffaa0bc4095"
format_stamp: "Formatted at 2026-09-27 23:04:32 on dist-test-slave-lmfp"
server_key: "26b8ace99dc94f955106d8b89cc0c7e2"
server_key_iv: "ddb07c82a631d101ba3f00bd1df6b151"
server_key_version: "encryptionkey@0"
I20260927 23:04:35.590276 10934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:35.601213 10934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:35.604652 10934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:35.604883 10934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:35.605705 10934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:35.606848 10934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:35.606973 10934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:35.607146 10934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:35.607244 10934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:35.644675 10934 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:36617
I20260927 23:04:35.644753 11136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:36617 every 8 connection(s)
I20260927 23:04:35.646054 10934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:35.656694 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 10934
I20260927 23:04:35.657068 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 10367
Sep 27 23:04:35 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550275, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
Sep 27 23:04:35 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550275, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
Sep 27 23:04:35 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550275, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260927 23:04:35.685194 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.651039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:60067 (local address 127.3.240.253:34805)
0927 23:04:35.665069 (+ 14030us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:35.665078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:35.665441 (+ 363us) server_negotiation.cc:407] Connection header received
0927 23:04:35.665619 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:35.665627 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:35.665729 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:35.665849 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:35.667425 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.668515 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.670485 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.671370 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.671510 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.677835 (+ 6325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:35.677844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:35.677852 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:35.677898 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:35.681459 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.682512 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.682518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.682521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.682672 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.683202 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.683207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.683210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.683533 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:35.683675 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:35.684562 (+ 887us) server_negotiation.cc:299] Negotiation successful
0927 23:04:35.684772 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13869,"thread_start_us":14046,"threads_started":1}
I20260927 23:04:35.686197 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.650822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.3.240.193:60067)
0927 23:04:35.665340 (+ 14518us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:35.665354 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:35.665492 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:35.666078 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:35.666083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:35.666811 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:35.667234 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.667252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.668677 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.668682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.670300 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.670316 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.671878 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.671887 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.672216 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.672987 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:35.673018 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:35.677540 (+ 4522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:35.681631 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:35.681640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:35.681655 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:35.682322 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:35.682803 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:35.682808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:35.682812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:35.683084 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:35.683659 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:35.683667 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:35.684070 (+ 403us) client_negotiation.cc:769] Sending connection context
0927 23:04:35.684462 (+ 392us) client_negotiation.cc:240] Negotiation successful
0927 23:04:35.685007 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14414,"mutex_wait_us":30,"thread_start_us":14300,"threads_started":1}
I20260927 23:04:35.687348 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.649716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.193:42695)
0927 23:04:35.655350 (+ 5634us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:35.655369 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:35.655497 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:35.666096 (+ 10599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:35.666104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:35.666920 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:35.667872 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.667893 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.669800 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.669808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.671828 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.671846 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.674112 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.674120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.674363 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.675020 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:35.675049 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:35.678417 (+ 3368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:35.682773 (+ 4356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:35.682780 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:35.682785 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:35.683380 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:35.683783 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:35.683788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:35.683792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:35.684076 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:35.684767 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:35.684773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:35.684951 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 23:04:35.685100 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 23:04:35.686915 (+ 1815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5534,"mutex_wait_us":488,"thread_start_us":5398,"threads_started":1}
I20260927 23:04:35.687855 11139 heartbeater.cc:347] Connected to a master server at 127.3.240.253:34805
I20260927 23:04:35.688184 11139 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:35.688328 11141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.648769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:42695 (local address 127.3.240.254:37357)
0927 23:04:35.665504 (+ 16735us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:35.665513 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:35.665538 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:04:35.665613 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:35.665620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:35.665718 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:35.665851 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:35.668121 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.669600 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.672754 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.673647 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.673798 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.678715 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:35.678724 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:35.678732 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:35.678786 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:35.682635 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.683544 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.683549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.683552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.683689 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.684204 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.684211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.684215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.684555 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:35.684699 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:35.687680 (+ 2981us) server_negotiation.cc:299] Negotiation successful
0927 23:04:35.687895 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16591,"thread_start_us":16500,"threads_started":1}
I20260927 23:04:35.689141 11138 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37357
I20260927 23:04:35.689189 11139 heartbeater.cc:511] Master 127.3.240.253:34805 requested a full tablet report, sending...
I20260927 23:04:35.689285 11138 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:35.689770 11138 heartbeater.cc:511] Master 127.3.240.254:37357 requested a full tablet report, sending...
I20260927 23:04:35.690863 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.648687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:48079 (local address 127.3.240.252:35827)
0927 23:04:35.648882 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:35.648888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:35.650238 (+ 1350us) server_negotiation.cc:407] Connection header received
0927 23:04:35.653299 (+ 3061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:35.653308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:35.653411 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:35.653506 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:35.665973 (+ 12467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.667159 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.672255 (+ 5096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.672968 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.673070 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.683966 (+ 10896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:35.683974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:35.683978 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:35.684017 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:35.687759 (+ 3742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.688609 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.688614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.688617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.688760 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.689230 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.689235 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.689239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.689564 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:35.689682 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:35.690280 (+ 598us) server_negotiation.cc:299] Negotiation successful
0927 23:04:35.690506 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:04:35.690981 10172 ts_manager.cc:194] Re-registered known tserver with Master: e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617)
I20260927 23:04:35.691810 11142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.649245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35827 (local address 127.3.240.193:48079)
0927 23:04:35.650086 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:35.650128 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:35.652977 (+ 2849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:35.654062 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:35.654077 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:35.654928 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:35.664277 (+ 9349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.664316 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.667660 (+ 3344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.667665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.669790 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:35.669803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.673523 (+ 3720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.673529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:35.673703 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.674682 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:35.674706 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:35.680466 (+ 5760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:35.687928 (+ 7462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:35.687933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:35.687936 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:35.688468 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:35.688907 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:35.688912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:35.688916 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:35.689098 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:35.689795 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:35.689799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:35.689970 (+ 171us) client_negotiation.cc:769] Sending connection context
0927 23:04:35.691100 (+ 1130us) client_negotiation.cc:240] Negotiation successful
0927 23:04:35.691415 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":1,"thread_start_us":555,"threads_started":1}
I20260927 23:04:35.691792 10770 ts_manager.cc:194] Registered new tserver with Master: e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617)
I20260927 23:04:35.692034 10172 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:42695
I20260927 23:04:35.692808 11137 heartbeater.cc:347] Connected to a master server at 127.3.240.252:35827
I20260927 23:04:35.692919 11137 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:35.693288 11137 heartbeater.cc:511] Master 127.3.240.252:35827 requested a full tablet report, sending...
I20260927 23:04:35.695138 10957 ts_manager.cc:194] Registered new tserver with Master: e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617)
Sep 27 23:04:35 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.194@KRBTEST.COM
I20260927 23:04:35.703306 11147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:35.658139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.3.240.194:36347)
0927 23:04:35.681035 (+ 22896us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:35.681045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:35.681076 (+ 31us) server_negotiation.cc:407] Connection header received
0927 23:04:35.681147 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:35.681156 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:35.681428 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:35.681620 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:35.685634 (+ 4014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.689964 (+ 4330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.691925 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:35.692745 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:35.692887 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:35.697838 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:35.697848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:35.697859 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:35.697893 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:35.700852 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.701487 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.701493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.701495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.701607 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:35.701935 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:35.701950 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:35.701952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:35.702248 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:35.702449 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:35.702713 (+ 264us) server_negotiation.cc:299] Negotiation successful
0927 23:04:35.702911 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22719,"thread_start_us":22568,"threads_started":1}
I20260927 23:04:35.704380 10490 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:35.785426 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 10367
I20260927 23:04:35.801138 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:36347
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:37357
--webserver_port=38853
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805,127.3.240.252:35827 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:36.121001 11149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:36.121258 11149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:36.121407 11149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:36.128536 11149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:36.128640 11149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:36.128819 11149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:36.138638 11149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:36347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=38853
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805,127.3.240.252:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:36.139966 11149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:36.141202 11149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:36.155062 11155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:36.155085 11149 server_base.cc:1034] running on GCE node
W20260927 23:04:36.155215 11157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:36.155572 11154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:36.156530 11149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:36.157859 11149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:36.159093 11149 hybrid_clock.cc:648] HybridClock initialized: now 1790550276159024 us; error 40 us; skew 500 ppm
I20260927 23:04:36.159448 11149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:04:36.160146 11158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: true } } }
I20260927 23:04:36.160586 11158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:36.162248 11161 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: attempting to promote NON_VOTER c5880468b1f946c9a4e3015789415f1d to VOTER
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550276, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:36.163240 11149 init.cc:377] Logged in from keytab as oryx/127.3.240.194@KRBTEST.COM (short username oryx)
I20260927 23:04:36.163178 11161 consensus_queue.cc:237] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } }
I20260927 23:04:36.165256 10967 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 LEARNER]: Refusing update from remote peer a56b7432116b47e8ac17984a7421f31d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 23:04:36.165457 10779 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Refusing update from remote peer a56b7432116b47e8ac17984a7421f31d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260927 23:04:36.165943 11149 webserver.cc:460] Webserver started at http://127.3.240.194:38853/ using document root <none> and password file <none>
I20260927 23:04:36.166121 11158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 23:04:36.166479 11162 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260927 23:04:36.166822 11149 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:36.166965 11149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:36.170526 11158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5880468b1f946c9a4e3015789415f1d (127.3.240.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:36.171201 10967 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5880468b1f946c9a4e3015789415f1d (127.3.240.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:36.175035 11162 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:36.175122 11161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:36.175484 11161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:36.175400 11162 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:36.177747 11171 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:36.178208 11171 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:04:36.180665 11161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:36.180871 11161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:36.181348 11149 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.012s sys 0.000s
I20260927 23:04:36.181586 10779 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c5880468b1f946c9a4e3015789415f1d (127.3.240.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:36.184504 11172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:36.185827 11173 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a56b7432116b47e8ac17984a7421f31d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:36.186076 11173 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:04:36.186221 11149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:36.186396 11149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "f16ad42ff909452a81b5044bcd3de35d"
format_stamp: "Formatted at 2026-09-27 23:04:33 on dist-test-slave-lmfp"
server_key: "044926f08841fce9e4fb860c868dc277"
server_key_iv: "ff57c10655fd9c8b1d78575f52812809"
server_key_version: "encryptionkey@0"
I20260927 23:04:36.187425 11149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:36.211274 11149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:36.215438 11149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:36.215662 11149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:36.216565 11149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:36.217756 11149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:36.217839 11149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:36.217913 11149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:36.217973 11149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:36.249684 11149 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:36347
I20260927 23:04:36.249785 11285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:36347 every 8 connection(s)
I20260927 23:04:36.251165 11149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:36.260408 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 11149
I20260927 23:04:36.260707 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 10500
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.195@KRBTEST.COM
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550276, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550276, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550276, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260927 23:04:36.308272 11290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.254186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52085 (local address 127.3.240.253:34805)
0927 23:04:36.269969 (+ 15783us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.269978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.282740 (+ 12762us) server_negotiation.cc:407] Connection header received
0927 23:04:36.285595 (+ 2855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.285603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.285702 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.285812 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.289192 (+ 3380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.290315 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.293509 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.294267 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.294451 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.300095 (+ 5644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.300099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.300103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.300173 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.303880 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.305224 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.305233 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.305239 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.305884 (+ 645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.306348 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.306352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.306354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.306607 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.306773 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.307652 (+ 879us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.307877 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15609,"thread_start_us":15792,"threads_started":1}
I20260927 23:04:36.310438 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.262130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.3.240.195:42997)
0927 23:04:36.279607 (+ 17477us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.279618 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.280411 (+ 793us) server_negotiation.cc:407] Connection header received
0927 23:04:36.280509 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.280522 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.280846 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.281054 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.285020 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.287241 (+ 2221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.290356 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.291775 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.291959 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.300935 (+ 8976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.300957 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.300980 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.301027 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.305097 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.306914 (+ 1817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.306924 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.306927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.307131 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.307618 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.307626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.307632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.308129 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.308435 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.309647 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.309945 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17279,"thread_start_us":17123,"threads_started":1}
I20260927 23:04:36.311837 10623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:36.311862 11292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.259525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:53821 (local address 127.3.240.254:37357)
0927 23:04:36.276171 (+ 16646us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.276182 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.283292 (+ 7110us) server_negotiation.cc:407] Connection header received
0927 23:04:36.285105 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.285111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.285199 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.285309 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.292907 (+ 7598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.294006 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.296601 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.297291 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.297415 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.304083 (+ 6668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.304090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.304094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.304149 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.308698 (+ 4549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.309620 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.309626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.309629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.309791 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.310235 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.310239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.310242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.310554 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.310695 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.311204 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.311431 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16453,"thread_start_us":16699,"threads_started":1}
I20260927 23:04:36.312012 11293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.259610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.194:53821)
0927 23:04:36.281805 (+ 22195us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.281825 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.284575 (+ 2750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.285830 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.285842 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.286851 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:36.288204 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.288232 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.294204 (+ 5972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.294212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.296441 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.296455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.297764 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.297772 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.298649 (+ 877us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.300192 (+ 1543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:36.300211 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:36.303863 (+ 3652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:36.308854 (+ 4991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.308858 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.308862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.309482 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.309938 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.309943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.309947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.310103 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.310756 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:36.310760 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:36.310896 (+ 136us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.311059 (+ 163us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.311316 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22076,"mutex_wait_us":1,"spinlock_wait_cycles":23424,"thread_start_us":21859,"threads_started":1}
I20260927 23:04:36.313256 11296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.282459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35827 (local address 127.3.240.194:48589)
0927 23:04:36.285042 (+ 2583us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.285062 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.285165 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.293888 (+ 8723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.293893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.294606 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:36.295102 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.295114 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.296596 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.296601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.298408 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.298422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.299613 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.299622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.299861 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.300489 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:36.300513 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:36.303813 (+ 3300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:36.309897 (+ 6084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.309905 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.309909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.310726 (+ 817us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.311193 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.311198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.311202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.311386 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.312317 (+ 931us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:36.312322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:36.312495 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.312595 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.312866 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2455,"mutex_wait_us":68,"thread_start_us":2768,"threads_started":1}
I20260927 23:04:36.313848 11287 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37357
I20260927 23:04:36.314245 11287 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:36.314376 11297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.282648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:48589 (local address 127.3.240.252:35827)
0927 23:04:36.293469 (+ 10821us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.293476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.293498 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:04:36.293556 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.293563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.293654 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.294822 (+ 1168us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.295282 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.296414 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.298567 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.299381 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.299513 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.304084 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.304091 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.304095 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.304137 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.309733 (+ 5596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.310875 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.310878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.310880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.311058 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.311546 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.311552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.311555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.312122 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.313607 (+ 1485us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.313763 (+ 156us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.314008 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10680,"thread_start_us":10467,"threads_started":1}
I20260927 23:04:36.315270 11291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.254894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.3.240.194:52085)
0927 23:04:36.277201 (+ 22307us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.277276 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.285419 (+ 8143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.286076 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.286081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.288726 (+ 2645us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:36.288980 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.288992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.290543 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.290549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.292700 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.292714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.294927 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.294938 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.295311 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.296070 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:36.296109 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:36.299860 (+ 3751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:36.304051 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.304062 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.304081 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.305090 (+ 1009us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.306019 (+ 929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.306023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.306025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.306207 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.306736 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:36.306745 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:36.307168 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.308574 (+ 1406us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.309224 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21815,"thread_start_us":25839,"threads_started":1}
I20260927 23:04:36.315385 11287 heartbeater.cc:511] Master 127.3.240.254:37357 requested a full tablet report, sending...
I20260927 23:04:36.316138 11288 heartbeater.cc:347] Connected to a master server at 127.3.240.253:34805
I20260927 23:04:36.316293 11288 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:36.316406 11286 heartbeater.cc:347] Connected to a master server at 127.3.240.252:35827
I20260927 23:04:36.316560 11286 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:36.316862 11288 heartbeater.cc:511] Master 127.3.240.253:34805 requested a full tablet report, sending...
I20260927 23:04:36.317142 11286 heartbeater.cc:511] Master 127.3.240.252:35827 requested a full tablet report, sending...
I20260927 23:04:36.318086 10770 ts_manager.cc:194] Registered new tserver with Master: f16ad42ff909452a81b5044bcd3de35d (127.3.240.194:36347)
I20260927 23:04:36.318251 10957 ts_manager.cc:194] Registered new tserver with Master: f16ad42ff909452a81b5044bcd3de35d (127.3.240.194:36347)
I20260927 23:04:36.318594 10172 ts_manager.cc:194] Re-registered known tserver with Master: f16ad42ff909452a81b5044bcd3de35d (127.3.240.194:36347)
I20260927 23:04:36.320065 10172 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:53821
I20260927 23:04:36.401636 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 10500
I20260927 23:04:36.417829 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:42997
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:37357
--webserver_port=33001
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805,127.3.240.252:35827 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:04:36.695379 11138 heartbeater.cc:503] Master 127.3.240.254:37357 was elected leader, sending a full tablet report...
W20260927 23:04:36.740904 11299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:36.741195 11299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:36.741353 11299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:36.748612 11299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:36.748762 11299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:36.748936 11299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:36.758891 11299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:42997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=33001
--tserver_master_addrs=127.3.240.254:37357,127.3.240.253:34805,127.3.240.252:35827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:36.760243 11299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:36.761617 11299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:36.775317 11304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:36.775243 11305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:36.775671 11307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:36.775858 11299 server_base.cc:1034] running on GCE node
I20260927 23:04:36.776510 11299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:36.777381 11299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:36.778602 11299 hybrid_clock.cc:648] HybridClock initialized: now 1790550276778549 us; error 39 us; skew 500 ppm
I20260927 23:04:36.778889 11299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550276, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:36.782032 11299 init.cc:377] Logged in from keytab as oryx/127.3.240.195@KRBTEST.COM (short username oryx)
I20260927 23:04:36.783640 11299 webserver.cc:460] Webserver started at http://127.3.240.195:33001/ using document root <none> and password file <none>
I20260927 23:04:36.784365 11299 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:36.784504 11299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:36.789906 11299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 23:04:36.792825 11314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:36.794202 11299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 23:04:36.794363 11299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "64ecde76b14f43b09665f78aa1275f04"
format_stamp: "Formatted at 2026-09-27 23:04:33 on dist-test-slave-lmfp"
server_key: "fea633f544d601ee57396405a3e6c1ed"
server_key_iv: "0975d3e631c6da599c6b1cad0fd8be58"
server_key_version: "encryptionkey@0"
I20260927 23:04:36.795357 11299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:36.819780 11299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:36.823961 11299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:36.824225 11299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:36.825145 11299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:36.826475 11299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:36.826598 11299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:36.826712 11299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:36.826797 11299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:36.858702 11299 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42997
I20260927 23:04:36.858783 11426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42997 every 8 connection(s)
I20260927 23:04:36.860160 11299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:36.867190 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 11299
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550276, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550276, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260927 23:04:36.897311 11434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.863809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.195:55479)
0927 23:04:36.873579 (+ 9770us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.873593 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.873676 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.876185 (+ 2509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.876197 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.877224 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:36.878262 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.878286 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.879467 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.879476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.880730 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.880742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.881671 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.881678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.881896 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.882632 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:36.882658 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:36.887598 (+ 4940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:36.892342 (+ 4744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.892349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.892364 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.892817 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.893271 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.893275 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.893279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.893505 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.894209 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:36.894216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:36.894644 (+ 428us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.894969 (+ 325us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.895626 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9637,"mutex_wait_us":103,"spinlock_wait_cycles":35200,"thread_start_us":9106,"threads_started":1}
I20260927 23:04:36.900099 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.863245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:55479 (local address 127.3.240.254:37357)
0927 23:04:36.875617 (+ 12372us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.875625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.875645 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:04:36.875714 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.875722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.875810 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.875943 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.878464 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.879319 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.880862 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.881458 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.881570 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.887784 (+ 6214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.887790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.887795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.887833 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.892238 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.892941 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.892950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.892956 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.893168 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.893609 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.893617 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.893623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.894089 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.894221 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.898877 (+ 4656us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.899694 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12240,"thread_start_us":12139,"threads_started":1}
I20260927 23:04:36.901921 11428 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37357
I20260927 23:04:36.902531 11428 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:36.903517 11428 heartbeater.cc:511] Master 127.3.240.254:37357 requested a full tablet report, sending...
I20260927 23:04:36.905563 10172 ts_manager.cc:194] Re-registered known tserver with Master: 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997)
I20260927 23:04:36.906649 10172 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:55479
Sep 27 23:04:36 dist-test-slave-lmfp krb5kdc[10127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550276, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260927 23:04:36.912717 11433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.863808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35827 (local address 127.3.240.195:48383)
0927 23:04:36.864886 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.864949 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.871256 (+ 6307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.876798 (+ 5542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.876802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.877440 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:36.878262 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.878286 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.884610 (+ 6324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.884617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.886352 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.886371 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.888642 (+ 2271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.888651 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.888838 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.889383 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:36.889403 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:36.891977 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:36.909598 (+ 17621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.909603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.909606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.910047 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.910861 (+ 814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.910866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.910869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.911055 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.911664 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:36.911669 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:36.911852 (+ 183us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.911994 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.912261 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":96,"thread_start_us":401,"threads_started":1}
I20260927 23:04:36.913978 11431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.863047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:48383 (local address 127.3.240.252:35827)
0927 23:04:36.876415 (+ 13368us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.876423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.876452 (+ 29us) server_negotiation.cc:407] Connection header received
0927 23:04:36.876503 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.876507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.876607 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.877191 (+ 584us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.879130 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.884455 (+ 5325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.887825 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.888445 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.889162 (+ 717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.901636 (+ 12474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.901641 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.901646 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.901680 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.906071 (+ 4391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.910242 (+ 4171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.910247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.910251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.910407 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.911217 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.911224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.911227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.911563 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.911681 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.913341 (+ 1660us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.913611 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13242,"thread_start_us":13154,"threads_started":1}
I20260927 23:04:36.915678 11427 heartbeater.cc:347] Connected to a master server at 127.3.240.252:35827
I20260927 23:04:36.915808 11427 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:36.916185 11427 heartbeater.cc:511] Master 127.3.240.252:35827 requested a full tablet report, sending...
I20260927 23:04:36.917359 10957 ts_manager.cc:194] Registered new tserver with Master: 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997)
I20260927 23:04:36.922860 11435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.865504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.3.240.195:35709)
0927 23:04:36.880364 (+ 14860us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.880386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.880538 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.882941 (+ 2403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.882945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.883544 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:36.883940 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.883958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.899900 (+ 15942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.899905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.901411 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.901434 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.906364 (+ 4930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.906370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.906538 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.907303 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:36.907328 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:36.912476 (+ 5148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:36.917273 (+ 4797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.917280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.917283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.917696 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.919099 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:36.919104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:36.919107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:36.919513 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:36.921838 (+ 2325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:36.921845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:36.922027 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.922163 (+ 136us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.922468 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14706,"thread_start_us":14571,"threads_started":1}
I20260927 23:04:36.923323 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.865591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:35709 (local address 127.3.240.253:34805)
0927 23:04:36.882519 (+ 16928us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.882528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.882552 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:04:36.882641 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.882649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.882748 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.882875 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.884085 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.885142 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.901564 (+ 16422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.902437 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.902587 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.913984 (+ 11397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.913991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.913997 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.914043 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.917138 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.917815 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.917821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.917824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.917947 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.921185 (+ 3238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.921189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.921193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.921539 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.921710 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.922287 (+ 577us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.922537 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16777,"thread_start_us":17431,"threads_started":1}
I20260927 23:04:36.924132 11429 heartbeater.cc:347] Connected to a master server at 127.3.240.253:34805
I20260927 23:04:36.924270 11429 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:36.924543 11429 heartbeater.cc:511] Master 127.3.240.253:34805 requested a full tablet report, sending...
I20260927 23:04:36.925630 10770 ts_manager.cc:194] Registered new tserver with Master: 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997)
I20260927 23:04:36.927912 11443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.886098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43782 (local address 127.3.240.253:34805)
0927 23:04:36.899208 (+ 13110us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.899216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.899239 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:04:36.899303 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.899310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.899396 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.899514 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.903983 (+ 4469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.905600 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.909096 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.910166 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.910316 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.921189 (+ 10873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.921195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.921199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.921228 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.924920 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.925795 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.925800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.925805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.925965 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.926378 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.926382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.926385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.926673 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.926924 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.927309 (+ 385us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.927526 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12302,"thread_start_us":12918,"threads_started":1}
I20260927 23:04:36.928701 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.894739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36536 (local address 127.3.240.252:35827)
0927 23:04:36.908434 (+ 13695us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.908442 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.908934 (+ 492us) server_negotiation.cc:407] Connection header received
0927 23:04:36.909068 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.909073 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.909138 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.909228 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.911954 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.913799 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.915809 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.916425 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.916522 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.921410 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.921414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.921417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.921440 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.925231 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.926014 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.926021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.926028 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.926163 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.926838 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.926843 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.926847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.927234 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.927447 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.928130 (+ 683us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.928317 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13547,"thread_start_us":16029,"threads_started":1}
I20260927 23:04:36.930791 11432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.885161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.3.240.254:37357)
0927 23:04:36.911262 (+ 26101us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.911269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.911287 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:04:36.911331 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.911336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.911408 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.911501 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:36.913897 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.914924 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.918076 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.918496 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.918877 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.923162 (+ 4285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:36.923168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:36.923172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:36.923210 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:36.927623 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.928887 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.928891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.928893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.929000 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:36.929437 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:36.929441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:36.929443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:36.929767 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:36.930006 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.930305 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.930451 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25735,"thread_start_us":28522,"threads_started":1}
I20260927 23:04:36.934064 10172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:36.940318 10172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:36.968236 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.957622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36617 (local address 127.0.0.1:39336)
0927 23:04:36.958736 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.958757 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.958913 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.960613 (+ 1700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.960621 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.960651 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:36.961004 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.961018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.962360 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.962368 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.964821 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.964836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.966900 (+ 2064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.966919 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.967143 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.967176 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.967270 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.967389 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":1911,"threads_started":1}
I20260927 23:04:36.969029 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.958598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39336 (local address 127.3.240.193:36617)
0927 23:04:36.960152 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.960160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.960193 (+ 33us) server_negotiation.cc:407] Connection header received
0927 23:04:36.960272 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.960278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.960461 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.960596 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:36.961180 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.962243 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.964982 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.966690 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.966860 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.967096 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.967337 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.967523 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1418,"thread_start_us":1322,"threads_started":1}
I20260927 23:04:36.971544 11455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.958807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42997 (local address 127.0.0.1:37554)
0927 23:04:36.959742 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.959762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.959880 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.964017 (+ 4137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.964021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.964044 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:36.964517 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.964529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.966287 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.966295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.968763 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.968778 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.970784 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.970790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.970931 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.970964 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.971081 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.971201 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":803,"thread_start_us":417,"threads_started":1}
I20260927 23:04:36.971553 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.959277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37554 (local address 127.3.240.195:42997)
0927 23:04:36.963551 (+ 4274us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.963557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.963580 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:04:36.963648 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.963653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.963832 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.964023 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:36.964716 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.966174 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.969378 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.970591 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.970725 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.970907 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.971116 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.971265 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4154,"thread_start_us":4012,"threads_started":1}
I20260927 23:04:36.972142 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.958118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:36347 (local address 127.0.0.1:50622)
0927 23:04:36.959535 (+ 1417us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:36.959554 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:36.959682 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:36.965106 (+ 5424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:36.965111 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:36.965133 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:36.965452 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.965465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.967371 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.967376 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.969896 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:36.969910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.971458 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.971464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:36.971665 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.971709 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 23:04:36.971768 (+ 59us) client_negotiation.cc:240] Negotiation successful
0927 23:04:36.971888 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"thread_start_us":368,"threads_started":1}
I20260927 23:04:36.972963 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:36.958610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50622 (local address 127.3.240.194:36347)
0927 23:04:36.964605 (+ 5995us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:36.964615 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:36.964639 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:04:36.964727 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:36.964732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:36.964889 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:36.965649 (+ 760us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:36.965820 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.967213 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.970104 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:36.971231 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:36.972361 (+ 1130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:36.972529 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:36.972583 (+ 54us) server_negotiation.cc:299] Negotiation successful
0927 23:04:36.972679 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5850,"thread_start_us":5747,"threads_started":1}
I20260927 23:04:36.973057 11072 tablet_service.cc:1467] Processing CreateTablet for tablet 050fb08051264bdb87544ac359a41dda (DEFAULT_TABLE table=test-table [id=1e64949035e444da83a007570754b21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:36.974486 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050fb08051264bdb87544ac359a41dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:36.975122 11362 tablet_service.cc:1467] Processing CreateTablet for tablet 050fb08051264bdb87544ac359a41dda (DEFAULT_TABLE table=test-table [id=1e64949035e444da83a007570754b21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:36.976111 11362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050fb08051264bdb87544ac359a41dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:36.981266 11221 tablet_service.cc:1467] Processing CreateTablet for tablet 050fb08051264bdb87544ac359a41dda (DEFAULT_TABLE table=test-table [id=1e64949035e444da83a007570754b21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:36.982407 11221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 050fb08051264bdb87544ac359a41dda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:36.985987 11461 tablet_bootstrap.cc:492] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: Bootstrap starting.
I20260927 23:04:36.989235 11461 tablet_bootstrap.cc:654] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:36.990617 11462 tablet_bootstrap.cc:492] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: Bootstrap starting.
I20260927 23:04:36.990607 11461 log.cc:824] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:36.994612 11462 tablet_bootstrap.cc:654] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:36.995837 11462 log.cc:824] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:36.997447 11461 tablet_bootstrap.cc:492] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: No bootstrap required, opened a new log
I20260927 23:04:36.997565 11464 tablet_bootstrap.cc:492] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Bootstrap starting.
I20260927 23:04:36.997689 11461 ts_tablet_manager.cc:1397] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260927 23:04:36.998968 11462 tablet_bootstrap.cc:492] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: No bootstrap required, opened a new log
I20260927 23:04:36.999269 11462 ts_tablet_manager.cc:1397] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260927 23:04:37.000386 11464 tablet_bootstrap.cc:654] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:37.001252 11464 log.cc:824] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:37.003501 11464 tablet_bootstrap.cc:492] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: No bootstrap required, opened a new log
I20260927 23:04:37.003742 11464 ts_tablet_manager.cc:1397] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 23:04:37.008122 11461 raft_consensus.cc:348] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.008554 11461 raft_consensus.cc:374] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:37.008664 11461 raft_consensus.cc:729] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fd87ed1cbf47b39818dffaa0bc4095, State: Initialized, Role: FOLLOWER
I20260927 23:04:37.009640 11461 consensus_queue.cc:260] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.009784 11462 raft_consensus.cc:348] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.010200 11462 raft_consensus.cc:374] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:37.010322 11462 raft_consensus.cc:729] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ecde76b14f43b09665f78aa1275f04, State: Initialized, Role: FOLLOWER
I20260927 23:04:37.010285 11464 raft_consensus.cc:348] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.010654 11464 raft_consensus.cc:374] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:37.010738 11464 raft_consensus.cc:729] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16ad42ff909452a81b5044bcd3de35d, State: Initialized, Role: FOLLOWER
I20260927 23:04:37.011113 11462 consensus_queue.cc:260] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.011309 11464 consensus_queue.cc:260] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.012771 11461 ts_tablet_manager.cc:1428] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20260927 23:04:37.014185 11464 ts_tablet_manager.cc:1428] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260927 23:04:37.015458 11428 heartbeater.cc:503] Master 127.3.240.254:37357 was elected leader, sending a full tablet report...
I20260927 23:04:37.015879 11287 heartbeater.cc:503] Master 127.3.240.254:37357 was elected leader, sending a full tablet report...
I20260927 23:04:37.019685 11462 ts_tablet_manager.cc:1428] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: Time spent starting tablet: real 0.020s user 0.008s sys 0.005s
I20260927 23:04:37.025488 11469 raft_consensus.cc:482] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:37.025739 11469 raft_consensus.cc:504] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.027041 11469 leader_election.cc:288] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617), 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997)
I20260927 23:04:37.033474 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.027356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:44197 (local address 127.3.240.193:36617)
0927 23:04:37.027574 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:37.027583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:37.028654 (+ 1071us) server_negotiation.cc:407] Connection header received
0927 23:04:37.028861 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:37.028864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:37.028922 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:37.029012 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:37.029539 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.030199 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.031938 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.032645 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.032817 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.032884 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:37.033140 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 23:04:37.033246 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:04:37.033486 11460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.027419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:34891 (local address 127.3.240.195:42997)
0927 23:04:37.027666 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:37.027674 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:37.028751 (+ 1077us) server_negotiation.cc:407] Connection header received
0927 23:04:37.028921 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:37.028925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:37.028988 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:37.029073 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:37.029600 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.030286 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.031965 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.032654 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.032846 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.032916 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:37.033170 (+ 254us) server_negotiation.cc:299] Negotiation successful
0927 23:04:37.033279 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 23:04:37.033629 11470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.027231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36617 (local address 127.3.240.194:44197)
0927 23:04:37.028555 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:37.028570 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:37.028729 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:37.029035 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:37.029040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:37.029060 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:37.029404 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:37.029415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.030326 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.030331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:37.031794 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:37.031805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.032851 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.032856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:37.033004 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.033024 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 23:04:37.033119 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 23:04:37.033227 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1200,"spinlock_wait_cycles":7296,"thread_start_us":547,"threads_started":1}
I20260927 23:04:37.033778 11471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.027231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42997 (local address 127.3.240.194:34891)
0927 23:04:37.028642 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:37.028656 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:37.028807 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:37.029096 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:37.029101 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:37.029141 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:37.029463 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:37.029470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.030417 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.030422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:37.031816 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:37.031827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.032903 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.032909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:37.033037 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.033056 (+ 19us) client_negotiation.cc:769] Sending connection context
0927 23:04:37.033149 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 23:04:37.033255 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"mutex_wait_us":58,"spinlock_wait_cycles":3712,"thread_start_us":619,"threads_started":1}
W20260927 23:04:37.034541 11289 tablet.cc:2369] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:04:37.034601 11092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050fb08051264bdb87544ac359a41dda" candidate_uuid: "f16ad42ff909452a81b5044bcd3de35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" is_pre_election: true
I20260927 23:04:37.034631 11382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050fb08051264bdb87544ac359a41dda" candidate_uuid: "f16ad42ff909452a81b5044bcd3de35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ecde76b14f43b09665f78aa1275f04" is_pre_election: true
I20260927 23:04:37.035120 11092 raft_consensus.cc:2454] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f16ad42ff909452a81b5044bcd3de35d in term 0.
I20260927 23:04:37.035138 11382 raft_consensus.cc:2454] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f16ad42ff909452a81b5044bcd3de35d in term 0.
I20260927 23:04:37.035732 11174 leader_election.cc:302] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64ecde76b14f43b09665f78aa1275f04, e4fd87ed1cbf47b39818dffaa0bc4095, f16ad42ff909452a81b5044bcd3de35d; no voters:
I20260927 23:04:37.036093 11469 raft_consensus.cc:2781] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:37.036237 11469 raft_consensus.cc:482] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:37.036329 11469 raft_consensus.cc:3037] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:37.039090 11469 raft_consensus.cc:504] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.039686 11469 leader_election.cc:288] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [CANDIDATE]: Term 1 election: Requested vote from peers e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617), 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997)
I20260927 23:04:37.040076 11092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050fb08051264bdb87544ac359a41dda" candidate_uuid: "f16ad42ff909452a81b5044bcd3de35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095"
I20260927 23:04:37.040129 11382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "050fb08051264bdb87544ac359a41dda" candidate_uuid: "f16ad42ff909452a81b5044bcd3de35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ecde76b14f43b09665f78aa1275f04"
I20260927 23:04:37.040248 11092 raft_consensus.cc:3037] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:37.040349 11382 raft_consensus.cc:3037] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:37.042326 11092 raft_consensus.cc:2454] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f16ad42ff909452a81b5044bcd3de35d in term 1.
I20260927 23:04:37.042467 11382 raft_consensus.cc:2454] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f16ad42ff909452a81b5044bcd3de35d in term 1.
I20260927 23:04:37.042707 11174 leader_election.cc:302] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4fd87ed1cbf47b39818dffaa0bc4095, f16ad42ff909452a81b5044bcd3de35d; no voters:
I20260927 23:04:37.043028 11469 raft_consensus.cc:2781] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:37.043514 11469 raft_consensus.cc:686] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 1 LEADER]: Becoming Leader. State: Replica: f16ad42ff909452a81b5044bcd3de35d, State: Running, Role: LEADER
I20260927 23:04:37.043988 11469 consensus_queue.cc:237] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } }
I20260927 23:04:37.048807 10171 catalog_manager.cc:5495] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d reported cstate change: term changed from 0 to 1, leader changed from <none> to f16ad42ff909452a81b5044bcd3de35d (127.3.240.194). New cstate: current_term: 1 leader_uuid: "f16ad42ff909452a81b5044bcd3de35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } health_report { overall_health: UNKNOWN } } }
I20260927 23:04:37.070118 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.065193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.3.240.194:36347)
0927 23:04:37.065354 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:37.065358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:37.065373 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:04:37.065445 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:37.065448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:37.065521 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:37.065630 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:37.065913 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.066584 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.068048 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.068611 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.068698 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.069146 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:37.069621 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:37.069718 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:37.069785 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 23:04:37.069839 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 23:04:37.077982 11092 raft_consensus.cc:1261] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 1 FOLLOWER]: Refusing update from remote peer f16ad42ff909452a81b5044bcd3de35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:37.077983 11382 raft_consensus.cc:1261] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 1 FOLLOWER]: Refusing update from remote peer f16ad42ff909452a81b5044bcd3de35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:37.079028 11472 consensus_queue.cc:1035] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:37.079351 11469 consensus_queue.cc:1035] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:37.092893 11475 mvcc.cc:205] Tried to move back new op lower bound from 7334093934901084160 to 7334093934775648256. Current Snapshot: MvccSnapshot[applied={T|T < 7334093934901084160}]
I20260927 23:04:37.114097 10171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:37.114329 10171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:37.120214 10171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a56b7432116b47e8ac17984a7421f31d: Sending DeleteTablet for 3 replicas of tablet 050fb08051264bdb87544ac359a41dda
I20260927 23:04:37.121872 11072 tablet_service.cc:1514] Processing DeleteTablet for tablet 050fb08051264bdb87544ac359a41dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:37 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39336
I20260927 23:04:37.122762 11484 tablet_replica.cc:330] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: stopping tablet replica
I20260927 23:04:37.123672 11221 tablet_service.cc:1514] Processing DeleteTablet for tablet 050fb08051264bdb87544ac359a41dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:37 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:50622
I20260927 23:04:37.124449 11484 raft_consensus.cc:2229] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:37.124437 11362 tablet_service.cc:1514] Processing DeleteTablet for tablet 050fb08051264bdb87544ac359a41dda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:37 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:37554
I20260927 23:04:37.124960 11484 raft_consensus.cc:2258] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:37.127127 11484 ts_tablet_manager.cc:1905] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:37.132515 11486 tablet_replica.cc:330] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: stopping tablet replica
I20260927 23:04:37.133141 11487 tablet_replica.cc:330] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: stopping tablet replica
I20260927 23:04:37.133824 11484 ts_tablet_manager.cc:1918] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:37.134497 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 10140
I20260927 23:04:37.134456 11487 raft_consensus.cc:2229] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:37.134909 11484 log.cc:1196] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/050fb08051264bdb87544ac359a41dda
I20260927 23:04:37.135519 11484 ts_tablet_manager.cc:1939] T 050fb08051264bdb87544ac359a41dda P e4fd87ed1cbf47b39818dffaa0bc4095: Deleting consensus metadata
I20260927 23:04:37.135636 10202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:37.139823 11486 raft_consensus.cc:2229] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:37.140468 11486 raft_consensus.cc:2258] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:37.140945 11487 raft_consensus.cc:2258] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:37.143142 11487 ts_tablet_manager.cc:1905] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:37.146163 11486 ts_tablet_manager.cc:1905] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:37.151335 11487 ts_tablet_manager.cc:1918] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:37.151486 11487 log.cc:1196] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/050fb08051264bdb87544ac359a41dda
I20260927 23:04:37.152144 11487 ts_tablet_manager.cc:1939] T 050fb08051264bdb87544ac359a41dda P 64ecde76b14f43b09665f78aa1275f04: Deleting consensus metadata
I20260927 23:04:37.154381 11443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.123135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.3.240.253:34805)
0927 23:04:37.123314 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:37.123320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:37.133459 (+ 10139us) server_negotiation.cc:407] Connection header received
0927 23:04:37.133980 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:37.133987 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:37.134081 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:37.134154 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:37.136584 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.138046 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.141292 (+ 3246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.142223 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.142346 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.146497 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:37.146503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:37.146507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:37.146546 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:37.150081 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:37.150931 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:37.150950 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:37.150954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:37.151561 (+ 607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:37.152177 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:37.152185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:37.152191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:37.152611 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:37.152890 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:37.153672 (+ 782us) server_negotiation.cc:299] Negotiation successful
0927 23:04:37.153986 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 23:04:37.158453 11486 ts_tablet_manager.cc:1918] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:37.158605 11486 log.cc:1196] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/050fb08051264bdb87544ac359a41dda
I20260927 23:04:37.159185 11486 ts_tablet_manager.cc:1939] T 050fb08051264bdb87544ac359a41dda P f16ad42ff909452a81b5044bcd3de35d: Deleting consensus metadata
I20260927 23:04:37.171998 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.133762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36550 (local address 127.3.240.252:35827)
0927 23:04:37.133959 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:37.133965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:37.154971 (+ 21006us) server_negotiation.cc:407] Connection header received
0927 23:04:37.155222 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:37.155228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:37.155312 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:37.155410 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:37.157797 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.158847 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.160802 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.161530 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.161626 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.165132 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:37.165137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:37.165142 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:37.165174 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:37.168871 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:37.169606 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:37.169612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:37.169615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:37.169755 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:37.170203 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:37.170210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:37.170214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:37.170594 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:37.170882 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:37.171412 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 23:04:37.171641 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 23:04:37.282250 10158 catalog_manager.cc:4841] TS f16ad42ff909452a81b5044bcd3de35d (127.3.240.194:36347): tablet 050fb08051264bdb87544ac359a41dda (table test-table [id=1e64949035e444da83a007570754b21d]) successfully deleted
I20260927 23:04:37.283627 10156 catalog_manager.cc:4841] TS e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617): tablet 050fb08051264bdb87544ac359a41dda (table test-table [id=1e64949035e444da83a007570754b21d]) successfully deleted
W20260927 23:04:37.283844 10156 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.3.240.193:36617, user_credentials={real_user=slave}} blocked reactor thread for 145347us
I20260927 23:04:37.284098 10158 catalog_manager.cc:4841] TS 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997): tablet 050fb08051264bdb87544ac359a41dda (table test-table [id=1e64949035e444da83a007570754b21d]) successfully deleted
I20260927 23:04:37.284552 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 10140
I20260927 23:04:37.336074 11446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.319325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.3.240.252:35827)
0927 23:04:37.319577 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:37.319586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:37.319611 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:04:37.319816 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:37.319823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:37.319902 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:37.320018 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:37.321736 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.323018 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.324988 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.325595 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.325696 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.329454 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:37.329463 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:37.329467 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:37.329524 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:37.333045 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:37.333891 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:37.333895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:37.333899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:37.334027 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:37.334488 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:37.334493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:37.334496 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:37.334840 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:37.335085 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:37.335469 (+ 384us) server_negotiation.cc:299] Negotiation successful
0927 23:04:37.335680 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260927 23:04:37.336923 11443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:37.317463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.3.240.253:34805)
0927 23:04:37.317661 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:37.317667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:37.319115 (+ 1448us) server_negotiation.cc:407] Connection header received
0927 23:04:37.319378 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:37.319385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:37.319479 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:37.319547 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:37.321874 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.323047 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.324591 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:37.325109 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:37.325236 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:37.329157 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:37.329170 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:37.329174 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:37.329229 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:37.332985 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:37.333766 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:37.333773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:37.333776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:37.333950 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:37.334400 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:37.334406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:37.334409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:37.334778 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:37.335079 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:37.336395 (+ 1316us) server_negotiation.cc:299] Negotiation successful
0927 23:04:37.336605 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 23:04:38.100090 11495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.099390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.193:52345)
0927 23:04:38.099837 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.099948 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":260,"threads_started":1}
W20260927 23:04:38.100538 11138 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:37357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
I20260927 23:04:38.103184 11496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.102472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.194:43523)
0927 23:04:38.102877 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.102995 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":232,"threads_started":1}
W20260927 23:04:38.103587 11287 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:37357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
I20260927 23:04:38.111790 11497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.110454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.195:38503)
0927 23:04:38.111453 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.111582 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":805,"threads_started":1}
W20260927 23:04:38.112215 11428 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:37357 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
I20260927 23:04:38.620328 11498 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a56b7432116b47e8ac17984a7421f31d)
I20260927 23:04:38.620579 11498 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } }
I20260927 23:04:38.621745 11498 leader_election.cc:288] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a56b7432116b47e8ac17984a7421f31d (127.3.240.254:37357), e15b5882405a4f9e97888ddd00fae3d6 (127.3.240.253:34805)
I20260927 23:04:38.622195 11499 raft_consensus.cc:482] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a56b7432116b47e8ac17984a7421f31d)
I20260927 23:04:38.622462 11499 raft_consensus.cc:504] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } }
I20260927 23:04:38.628602 11499 leader_election.cc:288] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a56b7432116b47e8ac17984a7421f31d (127.3.240.254:37357), c5880468b1f946c9a4e3015789415f1d (127.3.240.252:35827)
I20260927 23:04:38.629045 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.624248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43562)
0927 23:04:38.628532 (+ 4284us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.628709 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4180,"thread_start_us":4006,"threads_started":1}
W20260927 23:04:38.629364 10754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
I20260927 23:04:38.638432 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.622025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43550)
0927 23:04:38.637833 (+ 15808us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.638072 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15637,"threads_started":1}
W20260927 23:04:38.639094 10941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
I20260927 23:04:38.641105 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.634867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43576)
0927 23:04:38.640677 (+ 5810us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.640833 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5685,"thread_start_us":5541,"threads_started":1}
W20260927 23:04:38.641320 10754 leader_election.cc:334] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a56b7432116b47e8ac17984a7421f31d (127.3.240.254:37357): Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
I20260927 23:04:38.642242 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.641926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43580)
0927 23:04:38.642070 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.642145 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 23:04:38.642458 10941 leader_election.cc:334] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a56b7432116b47e8ac17984a7421f31d (127.3.240.254:37357): Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
I20260927 23:04:38.643730 11502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.622162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.3.240.253:34805)
0927 23:04:38.633990 (+ 11828us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:38.633997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:38.635453 (+ 1456us) server_negotiation.cc:407] Connection header received
0927 23:04:38.635533 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:38.635540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:38.635629 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:38.635729 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:38.636631 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.638184 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.641473 (+ 3289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.642728 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.642885 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.643024 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:38.643251 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 23:04:38.643374 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11691,"thread_start_us":11517,"threads_started":1}
I20260927 23:04:38.643730 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.622010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34805 (local address 127.0.0.1:43814)
0927 23:04:38.635167 (+ 13157us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.635182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:38.635315 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:38.635981 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:38.635988 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:38.636022 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:38.636352 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.636366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.638393 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.638402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.641263 (+ 2861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.641277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.642890 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.642894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.643067 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.643111 (+ 44us) client_negotiation.cc:769] Sending connection context
0927 23:04:38.643228 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 23:04:38.643352 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13039,"mutex_wait_us":1,"thread_start_us":12866,"threads_started":1}
I20260927 23:04:38.644367 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5880468b1f946c9a4e3015789415f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e15b5882405a4f9e97888ddd00fae3d6" is_pre_election: true
I20260927 23:04:38.644660 10779 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5880468b1f946c9a4e3015789415f1d in term 1.
I20260927 23:04:38.645152 10943 leader_election.cc:302] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5880468b1f946c9a4e3015789415f1d, e15b5882405a4f9e97888ddd00fae3d6; no voters: a56b7432116b47e8ac17984a7421f31d
I20260927 23:04:38.645531 11498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260927 23:04:38.645637 11498 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader a56b7432116b47e8ac17984a7421f31d)
I20260927 23:04:38.645718 11498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:04:38.646181 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.634264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35827 (local address 127.0.0.1:36572)
0927 23:04:38.634442 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.634467 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:38.634626 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:38.639705 (+ 5079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:38.639709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:38.639737 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:38.640003 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.640014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.642118 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.642124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.643959 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.643972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.645498 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.645504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.645681 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.645701 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 23:04:38.645803 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 23:04:38.645931 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260927 23:04:38.646265 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.635155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.3.240.252:35827)
0927 23:04:38.639471 (+ 4316us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:38.639475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:38.639488 (+ 13us) server_negotiation.cc:407] Connection header received
0927 23:04:38.639528 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:38.639531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:38.639577 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:38.639689 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:38.640632 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.641954 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.644158 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.645294 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.645483 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.645598 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:38.645822 (+ 224us) server_negotiation.cc:299] Negotiation successful
0927 23:04:38.645938 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":674,"server-negotiator.queue_time_us":4214,"thread_start_us":4093,"threads_started":1}
I20260927 23:04:38.647240 10967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e15b5882405a4f9e97888ddd00fae3d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c5880468b1f946c9a4e3015789415f1d" is_pre_election: true
I20260927 23:04:38.648667 11498 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } }
I20260927 23:04:38.649109 10967 raft_consensus.cc:2379] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e15b5882405a4f9e97888ddd00fae3d6 in current term 2: Already voted for candidate c5880468b1f946c9a4e3015789415f1d in this term.
I20260927 23:04:38.649350 11498 leader_election.cc:288] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [CANDIDATE]: Term 2 election: Requested vote from peers a56b7432116b47e8ac17984a7421f31d (127.3.240.254:37357), e15b5882405a4f9e97888ddd00fae3d6 (127.3.240.253:34805)
I20260927 23:04:38.649807 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.649357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43590)
0927 23:04:38.649592 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.649691 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260927 23:04:38.649923 10779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5880468b1f946c9a4e3015789415f1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e15b5882405a4f9e97888ddd00fae3d6"
I20260927 23:04:38.650059 10756 leader_election.cc:302] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e15b5882405a4f9e97888ddd00fae3d6; no voters: a56b7432116b47e8ac17984a7421f31d, c5880468b1f946c9a4e3015789415f1d
I20260927 23:04:38.650147 10779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:04:38.650799 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.650437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43602)
0927 23:04:38.650607 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.650681 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260927 23:04:38.651037 10941 leader_election.cc:334] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a56b7432116b47e8ac17984a7421f31d (127.3.240.254:37357): Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
I20260927 23:04:38.652371 10779 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5880468b1f946c9a4e3015789415f1d in term 2.
I20260927 23:04:38.652549 11499 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:04:38.652878 10943 leader_election.cc:302] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5880468b1f946c9a4e3015789415f1d, e15b5882405a4f9e97888ddd00fae3d6; no voters: a56b7432116b47e8ac17984a7421f31d
I20260927 23:04:38.653129 11498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:04:38.653492 11498 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [term 2 LEADER]: Becoming Leader. State: Replica: c5880468b1f946c9a4e3015789415f1d, State: Running, Role: LEADER
I20260927 23:04:38.653832 11498 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } }
I20260927 23:04:38.655543 11508 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5880468b1f946c9a4e3015789415f1d. Latest consensus state: current_term: 2 leader_uuid: "c5880468b1f946c9a4e3015789415f1d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:38.655825 11508 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:38.656355 11510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:38.659592 11510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:38.660295 11510 catalog_manager.cc:1259] Loaded cluster ID: 5d16037075ed46b783798781536f95e9
I20260927 23:04:38.660387 11510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:38.661830 11510 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:38.662576 11510 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Loaded TSK: 0
I20260927 23:04:38.663422 11510 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:04:38.723419 10956 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:38.730592 10956 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:38.734643 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.734244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43628)
0927 23:04:38.734419 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.734534 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260927 23:04:38.734802 10779 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [term 2 FOLLOWER]: Refusing update from remote peer c5880468b1f946c9a4e3015789415f1d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260927 23:04:38.735720 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.735294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43634)
0927 23:04:38.735459 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.735535 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 23:04:38.735636 11508 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260927 23:04:38.735941 10941 consensus_peers.cc:487] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d -> Peer a56b7432116b47e8ac17984a7421f31d (127.3.240.254:37357): Couldn't send request to peer a56b7432116b47e8ac17984a7421f31d. Status: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260927 23:04:38.738603 11499 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5880468b1f946c9a4e3015789415f1d. Latest consensus state: current_term: 2 leader_uuid: "c5880468b1f946c9a4e3015789415f1d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:38.738956 11499 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:04:38.741123 11498 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c5880468b1f946c9a4e3015789415f1d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:38.741492 11498 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:38.741521 11512 mvcc.cc:205] Tried to move back new op lower bound from 7334093941691031552 to 7334093941369110528. Current Snapshot: MvccSnapshot[applied={T|T < 7334093941691031552}]
I20260927 23:04:38.741707 11511 mvcc.cc:205] Tried to move back new op lower bound from 7334093941691031552 to 7334093941369110528. Current Snapshot: MvccSnapshot[applied={T|T < 7334093941691031552 or (T in {7334093941691031552})}]
I20260927 23:04:38.747560 11499 sys_catalog.cc:455] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c5880468b1f946c9a4e3015789415f1d" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "a56b7432116b47e8ac17984a7421f31d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37357 } } peers { permanent_uuid: "e15b5882405a4f9e97888ddd00fae3d6" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34805 } attrs { promote: false } } peers { permanent_uuid: "c5880468b1f946c9a4e3015789415f1d" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35827 } attrs { promote: false } } }
I20260927 23:04:38.747822 11499 sys_catalog.cc:458] T 00000000000000000000000000000000 P e15b5882405a4f9e97888ddd00fae3d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:04:38.759589 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.752749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.3.240.194:36347)
0927 23:04:38.753245 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:38.753256 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:38.753283 (+ 27us) server_negotiation.cc:407] Connection header received
0927 23:04:38.753360 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:38.753365 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:38.753451 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:38.754240 (+ 789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:38.754384 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.755544 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.757590 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.758441 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.758599 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.758712 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:38.759070 (+ 358us) server_negotiation.cc:299] Negotiation successful
0927 23:04:38.759233 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":249,"threads_started":1}
I20260927 23:04:38.760339 11501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.752700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:36347 (local address 127.0.0.1:50640)
0927 23:04:38.752964 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.752981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:38.753087 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:38.753624 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:38.753631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:38.753664 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:38.754092 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.754104 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.755714 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.755719 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.757424 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.757444 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.758674 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.758680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.758869 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.758894 (+ 25us) client_negotiation.cc:769] Sending connection context
0927 23:04:38.759954 (+ 1060us) client_negotiation.cc:240] Negotiation successful
0927 23:04:38.760109 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":50}
I20260927 23:04:38.761099 11221 tablet_service.cc:1467] Processing CreateTablet for tablet 5502819eef7044a48dfbd609d2ec0200 (DEFAULT_TABLE table=test-table [id=a2de974e0d114ff8a26d8d84b8dc3017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:38.761533 11221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502819eef7044a48dfbd609d2ec0200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:38.763590 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.752426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.3.240.193:36617)
0927 23:04:38.753567 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:38.753575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:38.754289 (+ 714us) server_negotiation.cc:407] Connection header received
0927 23:04:38.754531 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:38.754536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:38.754657 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:38.754768 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:38.755569 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.756975 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.760386 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.762005 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.762135 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.762274 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:38.763244 (+ 970us) server_negotiation.cc:299] Negotiation successful
0927 23:04:38.763359 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":874,"threads_started":1}
I20260927 23:04:38.764165 11520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.753505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36617 (local address 127.0.0.1:39348)
0927 23:04:38.754192 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.754211 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:38.754383 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:38.754806 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:38.754811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:38.754865 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:38.755321 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.755335 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.757181 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.757188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.760157 (+ 2969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.760177 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.762726 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.762735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.763036 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.763086 (+ 50us) client_negotiation.cc:769] Sending connection context
0927 23:04:38.763768 (+ 682us) client_negotiation.cc:240] Negotiation successful
0927 23:04:38.763907 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":4,"thread_start_us":595,"threads_started":1}
I20260927 23:04:38.765643 11072 tablet_service.cc:1467] Processing CreateTablet for tablet 5502819eef7044a48dfbd609d2ec0200 (DEFAULT_TABLE table=test-table [id=a2de974e0d114ff8a26d8d84b8dc3017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:38.766022 11072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502819eef7044a48dfbd609d2ec0200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:38.768839 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.751739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42997 (local address 127.0.0.1:37560)
0927 23:04:38.752040 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.752062 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:38.752180 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:38.763746 (+ 11566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:38.763754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:38.763779 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:38.763959 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.763968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.765411 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.765418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.766598 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:38.766608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.768198 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.768204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:38.768380 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.768413 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 23:04:38.768495 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 23:04:38.768610 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260927 23:04:38.769596 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.751876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.3.240.195:42997)
0927 23:04:38.763367 (+ 11491us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:38.763374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:38.763395 (+ 21us) server_negotiation.cc:407] Connection header received
0927 23:04:38.763475 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:38.763480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:38.763577 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:38.764094 (+ 517us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:38.764256 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.765278 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.766748 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.768028 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.768181 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.768287 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:38.769109 (+ 822us) server_negotiation.cc:299] Negotiation successful
0927 23:04:38.769287 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11369,"thread_start_us":11282,"threads_started":1}
I20260927 23:04:38.769794 11521 tablet_bootstrap.cc:492] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: Bootstrap starting.
I20260927 23:04:38.770217 11362 tablet_service.cc:1467] Processing CreateTablet for tablet 5502819eef7044a48dfbd609d2ec0200 (DEFAULT_TABLE table=test-table [id=a2de974e0d114ff8a26d8d84b8dc3017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:38.770622 11362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5502819eef7044a48dfbd609d2ec0200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:38.771693 11521 tablet_bootstrap.cc:654] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:38.775286 11524 tablet_bootstrap.cc:492] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: Bootstrap starting.
I20260927 23:04:38.775851 11523 tablet_bootstrap.cc:492] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: Bootstrap starting.
I20260927 23:04:38.777037 11524 tablet_bootstrap.cc:654] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:38.777515 11521 tablet_bootstrap.cc:492] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: No bootstrap required, opened a new log
I20260927 23:04:38.777604 11523 tablet_bootstrap.cc:654] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:38.777654 11521 ts_tablet_manager.cc:1397] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 23:04:38.778462 11521 raft_consensus.cc:348] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.778692 11521 raft_consensus.cc:374] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:38.778790 11521 raft_consensus.cc:729] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f16ad42ff909452a81b5044bcd3de35d, State: Initialized, Role: FOLLOWER
I20260927 23:04:38.779089 11521 consensus_queue.cc:260] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.780560 11521 ts_tablet_manager.cc:1428] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:38.781425 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.780231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.194:36493)
0927 23:04:38.781044 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.781214 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":585,"threads_started":1}
I20260927 23:04:38.782226 11523 tablet_bootstrap.cc:492] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: No bootstrap required, opened a new log
I20260927 23:04:38.782370 11523 ts_tablet_manager.cc:1397] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260927 23:04:38.782711 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.782186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.194:57885)
0927 23:04:38.782475 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.782565 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260927 23:04:38.783109 11524 tablet_bootstrap.cc:492] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: No bootstrap required, opened a new log
I20260927 23:04:38.783327 11524 ts_tablet_manager.cc:1397] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260927 23:04:38.783335 11286 heartbeater.cc:503] Master 127.3.240.252:35827 was elected leader, sending a full tablet report...
I20260927 23:04:38.783211 11523 raft_consensus.cc:348] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.783468 11523 raft_consensus.cc:374] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:38.783586 11523 raft_consensus.cc:729] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4fd87ed1cbf47b39818dffaa0bc4095, State: Initialized, Role: FOLLOWER
I20260927 23:04:38.783813 11523 consensus_queue.cc:260] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.784281 11524 raft_consensus.cc:348] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.784638 11524 raft_consensus.cc:374] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:38.784740 11523 ts_tablet_manager.cc:1428] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:38.784739 11524 raft_consensus.cc:729] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64ecde76b14f43b09665f78aa1275f04, State: Initialized, Role: FOLLOWER
I20260927 23:04:38.784961 11524 consensus_queue.cc:260] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.786875 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.785197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.193:56145)
0927 23:04:38.785729 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.785933 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"threads_started":1}
I20260927 23:04:38.787626 11137 heartbeater.cc:503] Master 127.3.240.252:35827 was elected leader, sending a full tablet report...
I20260927 23:04:38.788658 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.788239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.193:52819)
0927 23:04:38.788443 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.788532 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260927 23:04:38.789693 11524 ts_tablet_manager.cc:1428] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260927 23:04:38.791046 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.790147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.195:37553)
0927 23:04:38.790726 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.790814 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":319,"threads_started":1}
I20260927 23:04:38.791738 11427 heartbeater.cc:503] Master 127.3.240.252:35827 was elected leader, sending a full tablet report...
I20260927 23:04:38.792718 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.791644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.195:60095)
0927 23:04:38.792245 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.792452 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":300,"threads_started":1}
I20260927 23:04:38.817785 11527 raft_consensus.cc:482] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:38.817974 11527 raft_consensus.cc:504] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.818598 11527 leader_election.cc:288] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997), e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617)
I20260927 23:04:38.819180 11092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502819eef7044a48dfbd609d2ec0200" candidate_uuid: "f16ad42ff909452a81b5044bcd3de35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" is_pre_election: true
I20260927 23:04:38.819180 11382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502819eef7044a48dfbd609d2ec0200" candidate_uuid: "f16ad42ff909452a81b5044bcd3de35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ecde76b14f43b09665f78aa1275f04" is_pre_election: true
I20260927 23:04:38.819394 11382 raft_consensus.cc:2454] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f16ad42ff909452a81b5044bcd3de35d in term 0.
I20260927 23:04:38.819402 11092 raft_consensus.cc:2454] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f16ad42ff909452a81b5044bcd3de35d in term 0.
I20260927 23:04:38.819861 11176 leader_election.cc:302] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 64ecde76b14f43b09665f78aa1275f04, e4fd87ed1cbf47b39818dffaa0bc4095, f16ad42ff909452a81b5044bcd3de35d; no voters:
I20260927 23:04:38.820137 11527 raft_consensus.cc:2781] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:38.820255 11527 raft_consensus.cc:482] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:38.820354 11527 raft_consensus.cc:3037] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:38.822181 11527 raft_consensus.cc:504] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.822679 11527 leader_election.cc:288] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [CANDIDATE]: Term 1 election: Requested vote from peers 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997), e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617)
I20260927 23:04:38.823145 11382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502819eef7044a48dfbd609d2ec0200" candidate_uuid: "f16ad42ff909452a81b5044bcd3de35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64ecde76b14f43b09665f78aa1275f04"
I20260927 23:04:38.823146 11092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5502819eef7044a48dfbd609d2ec0200" candidate_uuid: "f16ad42ff909452a81b5044bcd3de35d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095"
I20260927 23:04:38.823336 11382 raft_consensus.cc:3037] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:38.823338 11092 raft_consensus.cc:3037] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:38.826104 11382 raft_consensus.cc:2454] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f16ad42ff909452a81b5044bcd3de35d in term 1.
I20260927 23:04:38.826104 11092 raft_consensus.cc:2454] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f16ad42ff909452a81b5044bcd3de35d in term 1.
I20260927 23:04:38.826493 11174 leader_election.cc:302] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4fd87ed1cbf47b39818dffaa0bc4095, f16ad42ff909452a81b5044bcd3de35d; no voters:
I20260927 23:04:38.826764 11527 raft_consensus.cc:2781] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:38.827200 11527 raft_consensus.cc:686] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 1 LEADER]: Becoming Leader. State: Replica: f16ad42ff909452a81b5044bcd3de35d, State: Running, Role: LEADER
I20260927 23:04:38.827479 11527 consensus_queue.cc:237] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } }
I20260927 23:04:38.828188 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.827797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.194:49203)
0927 23:04:38.827975 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.828059 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 23:04:38.830008 10956 catalog_manager.cc:5495] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d reported cstate change: term changed from 0 to 1, leader changed from <none> to f16ad42ff909452a81b5044bcd3de35d (127.3.240.194). New cstate: current_term: 1 leader_uuid: "f16ad42ff909452a81b5044bcd3de35d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f16ad42ff909452a81b5044bcd3de35d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36347 } health_report { overall_health: HEALTHY } } }
I20260927 23:04:38.854218 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.849888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.3.240.194:36347)
0927 23:04:38.850090 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:38.850095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:38.850113 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:04:38.850299 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:38.850307 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:38.850364 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:38.850466 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:38.850788 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.851485 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.852638 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.853085 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.853212 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.853587 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:38.853682 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:38.853781 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:38.853930 (+ 149us) server_negotiation.cc:299] Negotiation successful
0927 23:04:38.853995 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 23:04:38.856787 11382 raft_consensus.cc:1261] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 1 FOLLOWER]: Refusing update from remote peer f16ad42ff909452a81b5044bcd3de35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:38.856832 11092 raft_consensus.cc:1261] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 1 FOLLOWER]: Refusing update from remote peer f16ad42ff909452a81b5044bcd3de35d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:38.857465 11534 consensus_queue.cc:1035] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64ecde76b14f43b09665f78aa1275f04" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:38.857785 11527 consensus_queue.cc:1035] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4fd87ed1cbf47b39818dffaa0bc4095" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:38.859890 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.859462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.195:51035)
0927 23:04:38.859629 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.859724 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260927 23:04:38.863039 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.862523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.193:47569)
0927 23:04:38.862699 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.862810 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 23:04:38.869656 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.869014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.195:60131)
0927 23:04:38.869290 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.869422 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260927 23:04:38.875128 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.874672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.194:46321)
0927 23:04:38.874867 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.874968 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 23:04:38.878775 10956 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:38.879088 10956 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:38.886475 10956 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5880468b1f946c9a4e3015789415f1d: Sending DeleteTablet for 3 replicas of tablet 5502819eef7044a48dfbd609d2ec0200
I20260927 23:04:38.887683 11072 tablet_service.cc:1514] Processing DeleteTablet for tablet 5502819eef7044a48dfbd609d2ec0200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:38 UTC) from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:39348
I20260927 23:04:38.887809 11540 mvcc.cc:205] Tried to move back new op lower bound from 7334093942192365568 to 7334093942079934464. Current Snapshot: MvccSnapshot[applied={T|T < 7334093942192365568}]
I20260927 23:04:38.888170 11071 tablet_service.cc:1514] Processing DeleteTablet for tablet 5502819eef7044a48dfbd609d2ec0200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:38 UTC) from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:39348
I20260927 23:04:38.889189 11362 tablet_service.cc:1514] Processing DeleteTablet for tablet 5502819eef7044a48dfbd609d2ec0200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:38 UTC) from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:37560
I20260927 23:04:38.889369 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.888664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.194:54165)
0927 23:04:38.889084 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.889183 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313}
I20260927 23:04:38.889585 11221 tablet_service.cc:1514] Processing DeleteTablet for tablet 5502819eef7044a48dfbd609d2ec0200 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:38 UTC) from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:50640
I20260927 23:04:38.889688 11546 tablet_replica.cc:330] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: stopping tablet replica
I20260927 23:04:38.889866 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.888856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.3.240.193:58313)
0927 23:04:38.889625 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:38.889734 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661}
I20260927 23:04:38.891201 11548 tablet_replica.cc:330] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: stopping tablet replica
I20260927 23:04:38.891829 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 10934
I20260927 23:04:38.891376 11548 raft_consensus.cc:2229] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:38.893587 11546 raft_consensus.cc:2229] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:38.893870 11548 raft_consensus.cc:2258] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:38.894346 11547 tablet_replica.cc:330] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: stopping tablet replica
I20260927 23:04:38.894479 11546 raft_consensus.cc:2258] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:38.894523 11547 raft_consensus.cc:2229] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:38.894769 11547 raft_consensus.cc:2258] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:38.895576 11546 ts_tablet_manager.cc:1905] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:38.896346 11547 ts_tablet_manager.cc:1905] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:38.897681 11548 ts_tablet_manager.cc:1905] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:38.901659 11546 ts_tablet_manager.cc:1918] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:38.901768 11546 log.cc:1196] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/5502819eef7044a48dfbd609d2ec0200
I20260927 23:04:38.902187 11546 ts_tablet_manager.cc:1939] T 5502819eef7044a48dfbd609d2ec0200 P e4fd87ed1cbf47b39818dffaa0bc4095: Deleting consensus metadata
I20260927 23:04:38.902292 11547 ts_tablet_manager.cc:1918] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:38.902410 11547 log.cc:1196] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/5502819eef7044a48dfbd609d2ec0200
I20260927 23:04:38.902835 11547 ts_tablet_manager.cc:1939] T 5502819eef7044a48dfbd609d2ec0200 P 64ecde76b14f43b09665f78aa1275f04: Deleting consensus metadata
I20260927 23:04:38.903252 10941 catalog_manager.cc:4841] TS e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617): tablet 5502819eef7044a48dfbd609d2ec0200 (table test-table [id=a2de974e0d114ff8a26d8d84b8dc3017]) successfully deleted
I20260927 23:04:38.903717 11548 ts_tablet_manager.cc:1918] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:38.903815 10943 catalog_manager.cc:4841] TS 64ecde76b14f43b09665f78aa1275f04 (127.3.240.195:42997): tablet 5502819eef7044a48dfbd609d2ec0200 (table test-table [id=a2de974e0d114ff8a26d8d84b8dc3017]) successfully deleted
I20260927 23:04:38.903862 11548 log.cc:1196] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/5502819eef7044a48dfbd609d2ec0200
W20260927 23:04:38.904129 10941 catalog_manager.cc:4808] TS e4fd87ed1cbf47b39818dffaa0bc4095 (127.3.240.193:36617): delete failed for tablet 5502819eef7044a48dfbd609d2ec0200 because the tablet was not found. No further retry: Not found: Tablet not found: 5502819eef7044a48dfbd609d2ec0200
I20260927 23:04:38.904295 11548 ts_tablet_manager.cc:1939] T 5502819eef7044a48dfbd609d2ec0200 P f16ad42ff909452a81b5044bcd3de35d: Deleting consensus metadata
I20260927 23:04:38.905253 10943 catalog_manager.cc:4841] TS f16ad42ff909452a81b5044bcd3de35d (127.3.240.194:36347): tablet 5502819eef7044a48dfbd609d2ec0200 (table test-table [id=a2de974e0d114ff8a26d8d84b8dc3017]) successfully deleted
I20260927 23:04:38.910012 11518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:38.892799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.3.240.193:36617)
0927 23:04:38.892966 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:38.892974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:38.895498 (+ 2524us) server_negotiation.cc:407] Connection header received
0927 23:04:38.895560 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:38.895567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:38.895646 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:38.895737 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:38.897019 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.898034 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.900796 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:38.901479 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:38.901572 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:38.905537 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:38.905548 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:38.905562 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:38.905592 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:38.908022 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:38.908574 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:38.908581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:38.908583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:38.908693 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:38.908984 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:38.908986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:38.908988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:38.909292 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:38.909480 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:38.909663 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 23:04:38.909799 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 23:04:38.910595 11132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:38.997611 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 10934
I20260927 23:04:39.014335 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 11149
I20260927 23:04:39.025837 11519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:39.014986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50658 (local address 127.3.240.194:36347)
0927 23:04:39.015174 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:39.015178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:39.015213 (+ 35us) server_negotiation.cc:407] Connection header received
0927 23:04:39.015368 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:39.015371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:39.015442 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:39.015515 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:39.016654 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:39.017323 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:39.018434 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:39.018794 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:39.018878 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:39.021189 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:39.021201 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:39.021216 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:39.021241 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:39.023704 (+ 2463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:39.024234 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:39.024240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:39.024242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:39.024353 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:39.024657 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:39.024660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:39.024662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:39.024943 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:39.025133 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:39.025347 (+ 214us) server_negotiation.cc:299] Negotiation successful
0927 23:04:39.025495 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 23:04:39.026438 11281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:39.139775 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:39.139286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43644)
0927 23:04:39.139484 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:39.139589 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 23:04:39.140771 11500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:39.140410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37357 (local address 127.0.0.1:43650)
0927 23:04:39.140557 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:39.140655 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:37357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 23:04:39.145853 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 11149
I20260927 23:04:39.167972 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 11299
I20260927 23:04:39.180922 11517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:39.168776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37570 (local address 127.3.240.195:42997)
0927 23:04:39.168965 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:39.168969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:39.168986 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:04:39.169071 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:39.169075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:39.169139 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:39.169220 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:39.170626 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:39.171411 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:39.172548 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:39.173051 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:39.173189 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:39.175745 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:39.175770 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:39.175786 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:39.175829 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:39.178279 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:39.178897 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:39.178913 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:39.178916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:39.179103 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:39.179451 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:39.179456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:39.179460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:39.179804 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:39.180072 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:39.180361 (+ 289us) server_negotiation.cc:299] Negotiation successful
0927 23:04:39.180588 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 23:04:39.181559 11422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:39.269624 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 11299
I20260927 23:04:39.286321 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 10633
I20260927 23:04:39.287351 10800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:39.397262 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 10633
I20260927 23:04:39.416778 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 10734
I20260927 23:04:39.417757 10987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:39.532094 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 10734
2026-09-27T23:04:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8961 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:39 dist-test-slave-lmfp krb5kdc[11557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:39 dist-test-slave-lmfp krb5kdc[11557](info): set up 2 sockets
Sep 27 23:04:39 dist-test-slave-lmfp krb5kdc[11557](info): commencing operation
krb5kdc: starting...
W20260927 23:04:40.583824 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:40 dist-test-slave-lmfp krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.254.keytab.
Entry for principal oryx/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.254.keytab.
I20260927 23:04:40.773437 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39761
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.254.keytab
--principal=oryx/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:41.069295 11570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:41.069571 11570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:41.069733 11570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:41.076299 11570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:41.076445 11570 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260927 23:04:41.076539 11570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:41.076653 11570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:41.076747 11570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:41.076833 11570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:41.086166 11570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39761
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.254.keytab
--principal=oryx/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:41.087309 11570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:41.088495 11570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:41.099465 11578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:41.099540 11575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:41.099519 11576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:41.099975 11570 server_base.cc:1034] running on GCE node
I20260927 23:04:41.100662 11570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:41.101771 11570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:41.103053 11570 hybrid_clock.cc:648] HybridClock initialized: now 1790550281102955 us; error 51 us; skew 500 ppm
I20260927 23:04:41.103471 11570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:41 dist-test-slave-lmfp krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550281, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:41.106735 11570 init.cc:377] Logged in from keytab as oryx/127.3.240.254@KRBTEST.COM (short username oryx)
I20260927 23:04:41.109182 11570 webserver.cc:460] Webserver started at http://127.3.240.254:42821/ using document root <none> and password file <none>
I20260927 23:04:41.110029 11570 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:41.110140 11570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:41.110478 11570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:41.113809 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "2d25edc8ffc94b4ca88a23777036bf2b"
format_stamp: "Formatted at 2026-09-27 23:04:41 on dist-test-slave-lmfp"
server_key: "c017249df8b28f4480881a862876a042"
server_key_iv: "eb832b9e476711459804de74e8cb2cb8"
server_key_version: "encryptionkey@0"
I20260927 23:04:41.114526 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2d25edc8ffc94b4ca88a23777036bf2b"
format_stamp: "Formatted at 2026-09-27 23:04:41 on dist-test-slave-lmfp"
server_key: "c017249df8b28f4480881a862876a042"
server_key_iv: "eb832b9e476711459804de74e8cb2cb8"
server_key_version: "encryptionkey@0"
I20260927 23:04:41.119100 11570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260927 23:04:41.122167 11585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:41.123366 11570 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:41.123524 11570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "2d25edc8ffc94b4ca88a23777036bf2b"
format_stamp: "Formatted at 2026-09-27 23:04:41 on dist-test-slave-lmfp"
server_key: "c017249df8b28f4480881a862876a042"
server_key_iv: "eb832b9e476711459804de74e8cb2cb8"
server_key_version: "encryptionkey@0"
I20260927 23:04:41.123723 11570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:41.134073 11570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:41.137987 11570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:41.138242 11570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:41.159755 11570 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39761
I20260927 23:04:41.159760 11646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39761 every 8 connection(s)
I20260927 23:04:41.161172 11570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:41.164568 11647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:41.170761 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 11570
I20260927 23:04:41.171025 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:41.171715 4035 external_mini_cluster.cc:1428] Setting key ea3d0eb7d298a56eaaa230ac025c8a68
I20260927 23:04:41.173807 11647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: Bootstrap starting.
I20260927 23:04:41.176733 11647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:41.177791 11647 log.cc:824] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:41.180235 11647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: No bootstrap required, opened a new log
Sep 27 23:04:41 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:41.187582 11647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25edc8ffc94b4ca88a23777036bf2b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39761 } }
I20260927 23:04:41.187942 11647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:41.188064 11647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d25edc8ffc94b4ca88a23777036bf2b, State: Initialized, Role: FOLLOWER
I20260927 23:04:41.188709 11647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25edc8ffc94b4ca88a23777036bf2b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39761 } }
I20260927 23:04:41.188983 11647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:41.189124 11647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:41.189297 11647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:41.191439 11647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25edc8ffc94b4ca88a23777036bf2b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39761 } }
I20260927 23:04:41.191884 11647 leader_election.cc:302] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d25edc8ffc94b4ca88a23777036bf2b; no voters:
I20260927 23:04:41.192502 11647 leader_election.cc:288] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:41.192732 11652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:41.193720 11652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [term 1 LEADER]: Becoming Leader. State: Replica: 2d25edc8ffc94b4ca88a23777036bf2b, State: Running, Role: LEADER
I20260927 23:04:41.194707 11650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:41.173572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51182 (local address 127.3.240.254:39761)
0927 23:04:41.174380 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:41.174391 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:41.174430 (+ 39us) server_negotiation.cc:407] Connection header received
0927 23:04:41.176189 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:41.176208 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:41.176571 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:41.176940 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:41.178557 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.179930 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.181218 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.182028 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.182282 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:41.186418 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:41.186426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:41.186440 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:41.186482 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:41.189987 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:41.190678 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:41.190691 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:41.190702 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:41.190882 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:41.191297 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:41.191304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:41.191308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:41.191880 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:41.192206 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:41.192753 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 23:04:41.193199 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":403,"threads_started":1}
I20260927 23:04:41.194526 11652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25edc8ffc94b4ca88a23777036bf2b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39761 } }
I20260927 23:04:41.197031 11647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:41.201440 11653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d25edc8ffc94b4ca88a23777036bf2b. Latest consensus state: current_term: 1 leader_uuid: "2d25edc8ffc94b4ca88a23777036bf2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25edc8ffc94b4ca88a23777036bf2b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39761 } } }
I20260927 23:04:41.201905 11653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:41.204952 11654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d25edc8ffc94b4ca88a23777036bf2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d25edc8ffc94b4ca88a23777036bf2b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39761 } } }
I20260927 23:04:41.205257 11654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b [sys.catalog]: This master's current role is: LEADER
W20260927 23:04:41.207779 11667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:04:41.207903 11667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:04:41.208026 11666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:41.209980 11666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:41.217151 11666 catalog_manager.cc:1347] Generated new cluster ID: 6c45289ee1774819ac85aff504784f65
I20260927 23:04:41.217245 11666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:41.224845 11666 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:41.226423 11666 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:41.233161 11666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: Generated new TSK 0
I20260927 23:04:41.233809 11666 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
I20260927 23:04:41.287657 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39761
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:41.617529 11673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:41.617780 11673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:41.617894 11673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:41.624593 11673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:41.624699 11673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:41.624848 11673 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:04:41.624966 11673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:41.633747 11673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:39761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:41.634879 11673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:41.636219 11673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:41.648315 11678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:41.648803 11679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:41.648996 11681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:41.649513 11673 server_base.cc:1034] running on GCE node
I20260927 23:04:41.650017 11673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:41.650796 11673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:41.651989 11673 hybrid_clock.cc:648] HybridClock initialized: now 1790550281651899 us; error 63 us; skew 500 ppm
I20260927 23:04:41.652205 11673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:41 dist-test-slave-lmfp krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550281, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:41.654870 11673 init.cc:377] Logged in from keytab as oryx/127.3.240.193@KRBTEST.COM (short username oryx)
I20260927 23:04:41.656092 11673 webserver.cc:460] Webserver started at http://127.3.240.193:38561/ using document root <none> and password file <none>
I20260927 23:04:41.656733 11673 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:41.656806 11673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:41.657140 11673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:41.659819 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "c8cf5ee51c99463f9be1e92019449c31"
format_stamp: "Formatted at 2026-09-27 23:04:41 on dist-test-slave-lmfp"
server_key: "367dfcb295f5df777d2cfaf601743b7d"
server_key_iv: "368528048eb3a7c5124a726b107ec315"
server_key_version: "encryptionkey@0"
I20260927 23:04:41.660521 11673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "c8cf5ee51c99463f9be1e92019449c31"
format_stamp: "Formatted at 2026-09-27 23:04:41 on dist-test-slave-lmfp"
server_key: "367dfcb295f5df777d2cfaf601743b7d"
server_key_iv: "368528048eb3a7c5124a726b107ec315"
server_key_version: "encryptionkey@0"
I20260927 23:04:41.665198 11673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:04:41.668296 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:41.669411 11673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:04:41.669565 11673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "c8cf5ee51c99463f9be1e92019449c31"
format_stamp: "Formatted at 2026-09-27 23:04:41 on dist-test-slave-lmfp"
server_key: "367dfcb295f5df777d2cfaf601743b7d"
server_key_iv: "368528048eb3a7c5124a726b107ec315"
server_key_version: "encryptionkey@0"
I20260927 23:04:41.669762 11673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:41.695901 11673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:41.700141 11673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:41.700438 11673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:41.702832 11673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:41.702904 11673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:41.703079 11673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:41.703137 11673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 23:04:41 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550281, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:41.727941 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:41.703645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:44015 (local address 127.3.240.254:39761)
0927 23:04:41.705234 (+ 1589us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:41.705238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:41.706699 (+ 1461us) server_negotiation.cc:407] Connection header received
0927 23:04:41.707932 (+ 1233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:41.707940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:41.708019 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:41.708162 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:41.710801 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.711966 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.715411 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.716163 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.716305 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:41.721232 (+ 4927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:41.721236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:41.721239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:41.721266 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:41.724245 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:41.725128 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:41.725134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:41.725156 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:41.725298 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:41.725879 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:41.725882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:41.725883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:41.726176 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:41.726334 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:41.727275 (+ 941us) server_negotiation.cc:299] Negotiation successful
0927 23:04:41.727521 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1488,"thread_start_us":1405,"threads_started":1}
I20260927 23:04:41.729497 11698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:41.704228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39761 (local address 127.3.240.193:44015)
0927 23:04:41.705334 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:41.705380 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:41.707636 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:41.708614 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:41.708627 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:41.709525 (+ 898us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:41.710585 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:41.710621 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.712162 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.712171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:41.715224 (+ 3053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:41.715247 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.716433 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.716439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:41.716811 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:41.717530 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:41.717561 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:41.721025 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:41.724367 (+ 3342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:41.724375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:41.724388 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:41.725008 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:41.725418 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:41.725424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:41.725429 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:41.725732 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:41.726270 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:41.726280 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:41.726780 (+ 500us) client_negotiation.cc:769] Sending connection context
0927 23:04:41.727178 (+ 398us) client_negotiation.cc:240] Negotiation successful
0927 23:04:41.727763 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"thread_start_us":385,"threads_started":1}
I20260927 23:04:41.746279 11673 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:36051
I20260927 23:04:41.746315 11808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:36051 every 8 connection(s)
I20260927 23:04:41.747467 11673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:41.749528 11809 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39761
I20260927 23:04:41.750249 11809 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:41.750906 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:41.736201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:33915 (local address 127.3.240.254:39761)
0927 23:04:41.736356 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:41.736362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:41.737258 (+ 896us) server_negotiation.cc:407] Connection header received
0927 23:04:41.737408 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:41.737410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:41.737459 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:41.737536 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:41.739114 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.740075 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.741273 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.741815 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.741942 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:41.744678 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:41.744683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:41.744686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:41.744710 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:41.747431 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:41.748203 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:41.748208 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:41.748211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:41.748346 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:41.748837 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:41.748843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:41.748848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:41.749170 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:41.749355 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:41.750414 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0927 23:04:41.750626 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:04:41.751021 11809 heartbeater.cc:511] Master 127.3.240.254:39761 requested a full tablet report, sending...
I20260927 23:04:41.751474 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:41.736389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39761 (local address 127.3.240.193:33915)
0927 23:04:41.737139 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:41.737178 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:41.737307 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:41.737591 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:41.737597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:41.738249 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:41.738926 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:41.738940 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.740194 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.740200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:41.741140 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:41.741150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:41.742014 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:41.742023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:41.742257 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:41.743150 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:41.743185 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:41.744488 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:41.747542 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:41.747546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:41.747549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:41.748069 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:41.748462 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:41.748469 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:41.748475 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:41.748676 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:41.749620 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:41.749624 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:41.749806 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 23:04:41.749941 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 23:04:41.750228 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":408,"threads_started":1}
I20260927 23:04:41.753681 11602 ts_manager.cc:194] Registered new tserver with Master: c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051)
I20260927 23:04:41.755072 11602 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:44015
I20260927 23:04:41.757493 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 11673
I20260927 23:04:41.757695 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:41.758337 4035 external_mini_cluster.cc:1428] Setting key 1c57d698bfdff55d5706d0dc2b5e1157
WARNING: no policy specified for oryx/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
I20260927 23:04:41.799052 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39761
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:42.092280 11813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:42.092541 11813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:42.092687 11813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:42.099452 11813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:42.099570 11813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:42.099689 11813 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:04:42.099802 11813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:42.108714 11813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:39761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:42.109874 11813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:42.111176 11813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:42.123765 11818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:42.123807 11819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:42.123842 11821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:42.124538 11813 server_base.cc:1034] running on GCE node
I20260927 23:04:42.125085 11813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:42.125886 11813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:42.127140 11813 hybrid_clock.cc:648] HybridClock initialized: now 1790550282127073 us; error 30 us; skew 500 ppm
I20260927 23:04:42.127490 11813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:42 dist-test-slave-lmfp krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550282, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:42.130470 11813 init.cc:377] Logged in from keytab as oryx/127.3.240.194@KRBTEST.COM (short username oryx)
I20260927 23:04:42.131987 11813 webserver.cc:460] Webserver started at http://127.3.240.194:39073/ using document root <none> and password file <none>
I20260927 23:04:42.132669 11813 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:42.132803 11813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:42.133095 11813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:42.135804 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "0061e7ce1e494432a56f67927fc03b66"
format_stamp: "Formatted at 2026-09-27 23:04:42 on dist-test-slave-lmfp"
server_key: "7d5d4fe8dfabba61f8ed0cd807ec4ff1"
server_key_iv: "21c98aafcf35ba50580d198537e2fc1d"
server_key_version: "encryptionkey@0"
I20260927 23:04:42.136490 11813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "0061e7ce1e494432a56f67927fc03b66"
format_stamp: "Formatted at 2026-09-27 23:04:42 on dist-test-slave-lmfp"
server_key: "7d5d4fe8dfabba61f8ed0cd807ec4ff1"
server_key_iv: "21c98aafcf35ba50580d198537e2fc1d"
server_key_version: "encryptionkey@0"
I20260927 23:04:42.141006 11813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:04:42.144243 11828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:42.145438 11813 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:42.145581 11813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "0061e7ce1e494432a56f67927fc03b66"
format_stamp: "Formatted at 2026-09-27 23:04:42 on dist-test-slave-lmfp"
server_key: "7d5d4fe8dfabba61f8ed0cd807ec4ff1"
server_key_iv: "21c98aafcf35ba50580d198537e2fc1d"
server_key_version: "encryptionkey@0"
I20260927 23:04:42.145795 11813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:42.167630 11813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:42.172511 11813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:42.172791 11813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:42.175887 11813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:42.175983 11813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:42.176101 11813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:42.176159 11813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 23:04:42 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550282, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:42.200131 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:42.175964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:56039 (local address 127.3.240.254:39761)
0927 23:04:42.176150 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:42.176155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:42.177609 (+ 1454us) server_negotiation.cc:407] Connection header received
0927 23:04:42.179642 (+ 2033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:42.179648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:42.179750 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:42.179847 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:42.181973 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.182941 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.186341 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.186860 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.187031 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:42.192471 (+ 5440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:42.192480 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:42.192486 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:42.192541 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:42.195976 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:42.196940 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:42.196944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:42.196946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:42.197102 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:42.197732 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:42.197737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:42.197740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:42.198161 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:42.198293 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:42.199286 (+ 993us) server_negotiation.cc:299] Negotiation successful
0927 23:04:42.199628 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260927 23:04:42.200794 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:42.176544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39761 (local address 127.3.240.194:56039)
0927 23:04:42.177446 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:42.177493 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:42.179378 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:42.180148 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:42.180161 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:42.180809 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:42.181732 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:42.181766 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.183134 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.183142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:42.186147 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:42.186172 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.187173 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.187181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:42.187521 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:42.188360 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:42.188387 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:42.192077 (+ 3690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:42.196148 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:42.196159 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:42.196178 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:42.196806 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:42.197268 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:42.197275 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:42.197283 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:42.197605 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:42.198270 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:42.198278 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:42.198750 (+ 472us) client_negotiation.cc:769] Sending connection context
0927 23:04:42.199167 (+ 417us) client_negotiation.cc:240] Negotiation successful
0927 23:04:42.199612 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":371,"threads_started":1}
I20260927 23:04:42.224650 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:42.209597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42229 (local address 127.3.240.254:39761)
0927 23:04:42.209771 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:42.209778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:42.210416 (+ 638us) server_negotiation.cc:407] Connection header received
0927 23:04:42.210604 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:42.210609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:42.210676 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:42.210767 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:42.212180 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.213040 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.215577 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.216129 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.216260 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:42.218695 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:42.218699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:42.218702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:42.218729 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:42.221881 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:42.222747 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:42.222755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:42.222759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:42.222892 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:42.223264 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:42.223271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:42.223274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:42.223584 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:42.223740 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:42.224106 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 23:04:42.224315 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:04:42.224783 11902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:42.209686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39761 (local address 127.3.240.194:42229)
0927 23:04:42.210323 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:42.210339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:42.210483 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:42.210787 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:42.210791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:42.211535 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:42.211963 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:42.211982 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.213196 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.213202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:42.215434 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:42.215452 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.216329 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.216338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:42.216589 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:42.217213 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:42.217242 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:42.218512 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:42.221998 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:42.222004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:42.222008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:42.222618 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:42.223028 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:42.223033 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:42.223039 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:42.223164 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:42.223697 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:42.223702 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:42.223876 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 23:04:42.224002 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 23:04:42.224303 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":396,"threads_started":1}
I20260927 23:04:42.232553 11813 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:33961
I20260927 23:04:42.232645 11947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:33961 every 8 connection(s)
I20260927 23:04:42.233799 11813 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:42.235457 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 11813
I20260927 23:04:42.235611 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:42.235961 11948 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39761
I20260927 23:04:42.236156 4035 external_mini_cluster.cc:1428] Setting key 577765c2f581904bd2c726f22dc665db
I20260927 23:04:42.236224 11948 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:42.236917 11948 heartbeater.cc:511] Master 127.3.240.254:39761 requested a full tablet report, sending...
I20260927 23:04:42.238384 11602 ts_manager.cc:194] Registered new tserver with Master: 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961)
I20260927 23:04:42.239269 11602 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:56039
WARNING: no policy specified for oryx/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
I20260927 23:04:42.282142 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39761
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:42.587122 11952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:42.587396 11952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:42.587496 11952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:42.594543 11952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:42.594635 11952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:42.594712 11952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260927 23:04:42.594789 11952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:42.604061 11952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:39761
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:42.605175 11952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:42.606361 11952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:42.619246 11958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:42.619289 11957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:42.619171 11960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:42.620025 11952 server_base.cc:1034] running on GCE node
I20260927 23:04:42.620574 11952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:42.621379 11952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:42.622597 11952 hybrid_clock.cc:648] HybridClock initialized: now 1790550282622540 us; error 33 us; skew 500 ppm
I20260927 23:04:42.622879 11952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:42 dist-test-slave-lmfp krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550282, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:42.625770 11952 init.cc:377] Logged in from keytab as oryx/127.3.240.195@KRBTEST.COM (short username oryx)
I20260927 23:04:42.627280 11952 webserver.cc:460] Webserver started at http://127.3.240.195:39681/ using document root <none> and password file <none>
I20260927 23:04:42.627959 11952 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:42.628063 11952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:42.628373 11952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:42.631182 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "8554dbb06e544027bdd8ede9b87f5cb2"
format_stamp: "Formatted at 2026-09-27 23:04:42 on dist-test-slave-lmfp"
server_key: "e322f2fa62cda87240a81a3658e36f67"
server_key_iv: "6da32db5bdb1469d9f06fe6e087809bc"
server_key_version: "encryptionkey@0"
I20260927 23:04:42.631991 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "8554dbb06e544027bdd8ede9b87f5cb2"
format_stamp: "Formatted at 2026-09-27 23:04:42 on dist-test-slave-lmfp"
server_key: "e322f2fa62cda87240a81a3658e36f67"
server_key_iv: "6da32db5bdb1469d9f06fe6e087809bc"
server_key_version: "encryptionkey@0"
I20260927 23:04:42.636690 11952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:04:42.639830 11967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:42.641002 11952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:04:42.641155 11952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "8554dbb06e544027bdd8ede9b87f5cb2"
format_stamp: "Formatted at 2026-09-27 23:04:42 on dist-test-slave-lmfp"
server_key: "e322f2fa62cda87240a81a3658e36f67"
server_key_iv: "6da32db5bdb1469d9f06fe6e087809bc"
server_key_version: "encryptionkey@0"
I20260927 23:04:42.641352 11952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:42.667865 11952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:42.671940 11952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:42.672148 11952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:42.674726 11952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:42.674837 11952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:42.674974 11952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:42.675102 11952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 27 23:04:42 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550282, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:42.701462 11976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:42.676046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39761 (local address 127.3.240.195:53447)
0927 23:04:42.676864 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:42.676913 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:42.679129 (+ 2216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:42.680210 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:42.680235 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:42.681013 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:42.681762 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:42.681783 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.683200 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.683211 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:42.686187 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:42.686204 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.687281 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.687292 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:42.687765 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:42.688538 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:42.688571 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:42.692713 (+ 4142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:42.696867 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:42.696877 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:42.696892 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:42.697574 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:42.698047 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:42.698052 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:42.698056 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:42.698260 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:42.698860 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:42.698868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:42.699273 (+ 405us) client_negotiation.cc:769] Sending connection context
0927 23:04:42.699674 (+ 401us) client_negotiation.cc:240] Negotiation successful
0927 23:04:42.700205 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":380,"threads_started":1}
I20260927 23:04:42.702459 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:42.675517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:53447 (local address 127.3.240.254:39761)
0927 23:04:42.675732 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:42.675737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:42.677011 (+ 1274us) server_negotiation.cc:407] Connection header received
0927 23:04:42.679469 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:42.679480 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:42.679574 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:42.679657 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:42.681995 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.683025 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.686437 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.686919 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.687144 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:42.692989 (+ 5845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:42.692998 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:42.693005 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:42.693051 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:42.696619 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:42.697708 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:42.697715 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:42.697719 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:42.697915 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:42.698387 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:42.698394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:42.698398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:42.698711 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:42.698873 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:42.701999 (+ 3126us) server_negotiation.cc:299] Negotiation successful
0927 23:04:42.702176 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 23:04:42.722079 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:42.707607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:55845 (local address 127.3.240.254:39761)
0927 23:04:42.707798 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:42.707805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:42.708528 (+ 723us) server_negotiation.cc:407] Connection header received
0927 23:04:42.708658 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:42.708665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:42.708768 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:42.708963 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:42.709837 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.711191 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.712573 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.713274 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.713467 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:42.716062 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:42.716067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:42.716070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:42.716097 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:42.719364 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:42.720035 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:42.720043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:42.720046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:42.720166 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:42.720532 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:42.720540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:42.720543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:42.720897 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:42.721048 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:42.721490 (+ 442us) server_negotiation.cc:299] Negotiation successful
0927 23:04:42.721704 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 23:04:42.722198 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:42.707768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39761 (local address 127.3.240.195:55845)
0927 23:04:42.708420 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:42.708439 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:42.708554 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:42.708989 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:42.708993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:42.709408 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:42.709688 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:42.709697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.711353 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.711359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:42.712441 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:42.712453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.713548 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.713560 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:42.713752 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:42.714418 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:42.714461 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:42.715830 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:42.719501 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:42.719512 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:42.719516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:42.719915 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:42.720280 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:42.720285 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:42.720289 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:42.720423 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:42.721015 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:42.721020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:42.721241 (+ 221us) client_negotiation.cc:769] Sending connection context
0927 23:04:42.721391 (+ 150us) client_negotiation.cc:240] Negotiation successful
0927 23:04:42.721700 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":338,"threads_started":1}
I20260927 23:04:42.729585 11952 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33901
I20260927 23:04:42.729620 12086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33901 every 8 connection(s)
I20260927 23:04:42.730640 11952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:42.731142 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 11952
I20260927 23:04:42.731352 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:42.731833 4035 external_mini_cluster.cc:1428] Setting key c908d8d048e782586a82301c72c9454d
I20260927 23:04:42.732785 12087 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39761
I20260927 23:04:42.733230 12087 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:42.734138 12087 heartbeater.cc:511] Master 127.3.240.254:39761 requested a full tablet report, sending...
I20260927 23:04:42.735773 11602 ts_manager.cc:194] Registered new tserver with Master: 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901)
I20260927 23:04:42.736550 11602 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:53447
I20260927 23:04:42.746785 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:04:42.757285 11809 heartbeater.cc:503] Master 127.3.240.254:39761 was elected leader, sending a full tablet report...
Sep 27 23:04:42 dist-test-slave-lmfp krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550282, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 27 23:04:42 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:42 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:43.242368 11948 heartbeater.cc:503] Master 127.3.240.254:39761 was elected leader, sending a full tablet report...
I20260927 23:04:43.738597 12087 heartbeater.cc:503] Master 127.3.240.254:39761 was elected leader, sending a full tablet report...
W20260927 23:04:43.826004 12095 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.036s user 0.001s sys 0.013s
W20260927 23:04:43.827296 11697 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:42.784912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.3.240.254:39761)
0927 23:04:42.785149 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:42.785156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:42.785383 (+ 227us) server_negotiation.cc:407] Connection header received
0927 23:04:42.785582 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:42.785588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:42.785655 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:42.785768 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:42.786961 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.787700 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.788947 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:42.789361 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:42.789454 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.826879 (+1037425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51196: BlockingRecv error: recv got EOF from 127.0.0.1:51196 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 23:04:43 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550282, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:43.847971 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.830531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51202 (local address 127.3.240.254:39761)
0927 23:04:43.830734 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:43.830740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:43.830961 (+ 221us) server_negotiation.cc:407] Connection header received
0927 23:04:43.831170 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:43.831174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:43.831246 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:43.831326 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:43.832514 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.833203 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.834318 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.834651 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.834736 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.843220 (+ 8484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:43.843224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:43.843227 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:43.843254 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:43.845532 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:43.846173 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:43.846180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:43.846183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:43.846297 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:43.846665 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:43.846670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:43.846673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:43.846928 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:43.847249 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:43.847499 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 23:04:43.847685 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:04:43.851910 11602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:43.858464 11602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:43.890244 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.870449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.3.240.195:33901)
0927 23:04:43.882390 (+ 11941us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:43.882397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:43.883013 (+ 616us) server_negotiation.cc:407] Connection header received
0927 23:04:43.883283 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:43.883289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:43.883487 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:43.883679 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:43.884509 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.885738 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.888211 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.889146 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.889302 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.889522 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:43.889771 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 23:04:43.889926 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11776,"thread_start_us":11661,"threads_started":1}
I20260927 23:04:43.890419 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.869766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:33961 (local address 127.0.0.1:41868)
0927 23:04:43.880928 (+ 11162us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:43.880944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:43.881050 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:43.882846 (+ 1796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:43.882853 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:43.882872 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:43.883331 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.883343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.885300 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.885308 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.887547 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.887561 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.889553 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.889558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.889739 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.889771 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 23:04:43.889840 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 23:04:43.889958 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11049,"mutex_wait_us":52,"thread_start_us":10004,"threads_started":1}
I20260927 23:04:43.890765 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.870170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33901 (local address 127.0.0.1:39596)
0927 23:04:43.882861 (+ 12691us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:43.882879 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:43.883055 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:43.883806 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:43.883810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:43.883830 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:43.884240 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.884252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.885880 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.885890 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.888069 (+ 2179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.888079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.889326 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.889332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.889535 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.889589 (+ 54us) client_negotiation.cc:769] Sending connection context
0927 23:04:43.889734 (+ 145us) client_negotiation.cc:240] Negotiation successful
0927 23:04:43.890040 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12579,"thread_start_us":10947,"threads_started":1}
I20260927 23:04:43.890972 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.869862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45062 (local address 127.3.240.193:36051)
0927 23:04:43.881787 (+ 11925us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:43.881799 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:43.881828 (+ 29us) server_negotiation.cc:407] Connection header received
0927 23:04:43.881919 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:43.881926 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:43.882209 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:43.883115 (+ 906us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:43.883311 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.884869 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.887892 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.889261 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.889763 (+ 502us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.889976 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:43.890523 (+ 547us) server_negotiation.cc:299] Negotiation successful
0927 23:04:43.890694 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11724,"thread_start_us":11534,"threads_started":1}
I20260927 23:04:43.891480 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.870363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41868 (local address 127.3.240.194:33961)
0927 23:04:43.882098 (+ 11735us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:43.882109 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:43.882140 (+ 31us) server_negotiation.cc:407] Connection header received
0927 23:04:43.882229 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:43.882235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:43.882502 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:43.883551 (+ 1049us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:43.883755 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.885131 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.887762 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.889261 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.890703 (+ 1442us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.890898 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:43.890964 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 23:04:43.891194 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11556,"thread_start_us":11400,"threads_started":1}
I20260927 23:04:43.891616 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.869722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36051 (local address 127.0.0.1:45062)
0927 23:04:43.880164 (+ 10442us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:43.880202 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:43.880368 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:43.882457 (+ 2089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:43.882468 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:43.882505 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:43.883109 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.883126 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.885031 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.885037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.887265 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.887279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.889541 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.889547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.890032 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.890070 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 23:04:43.890145 (+ 75us) client_negotiation.cc:240] Negotiation successful
0927 23:04:43.890268 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10178,"mutex_wait_us":355,"thread_start_us":12873,"threads_started":1}
I20260927 23:04:43.894856 11739 tablet_service.cc:1467] Processing CreateTablet for tablet 360f6d826ac6471298193ab67898ed5c (DEFAULT_TABLE table=test-table [id=2077c09e0001407abe3665826b617799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:43.894892 11878 tablet_service.cc:1467] Processing CreateTablet for tablet 360f6d826ac6471298193ab67898ed5c (DEFAULT_TABLE table=test-table [id=2077c09e0001407abe3665826b617799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:43.894894 12017 tablet_service.cc:1467] Processing CreateTablet for tablet 360f6d826ac6471298193ab67898ed5c (DEFAULT_TABLE table=test-table [id=2077c09e0001407abe3665826b617799]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:43.896140 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360f6d826ac6471298193ab67898ed5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:43.896185 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360f6d826ac6471298193ab67898ed5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:43.896195 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 360f6d826ac6471298193ab67898ed5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:43.907083 12112 tablet_bootstrap.cc:492] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Bootstrap starting.
I20260927 23:04:43.910562 12112 tablet_bootstrap.cc:654] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:43.911666 12113 tablet_bootstrap.cc:492] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Bootstrap starting.
I20260927 23:04:43.911836 12112 log.cc:824] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:43.913179 12114 tablet_bootstrap.cc:492] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Bootstrap starting.
I20260927 23:04:43.915743 12114 tablet_bootstrap.cc:654] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:43.915740 12113 tablet_bootstrap.cc:654] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:43.916698 12114 log.cc:824] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:43.917379 12113 log.cc:824] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:43.919476 12114 tablet_bootstrap.cc:492] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: No bootstrap required, opened a new log
I20260927 23:04:43.919736 12114 ts_tablet_manager.cc:1397] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260927 23:04:43.920295 12113 tablet_bootstrap.cc:492] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: No bootstrap required, opened a new log
I20260927 23:04:43.920554 12113 ts_tablet_manager.cc:1397] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 23:04:43.921353 12112 tablet_bootstrap.cc:492] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: No bootstrap required, opened a new log
I20260927 23:04:43.921634 12112 ts_tablet_manager.cc:1397] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260927 23:04:43.926563 12114 raft_consensus.cc:348] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.926900 12114 raft_consensus.cc:374] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:43.927065 12114 raft_consensus.cc:729] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8554dbb06e544027bdd8ede9b87f5cb2, State: Initialized, Role: FOLLOWER
I20260927 23:04:43.927709 12114 consensus_queue.cc:260] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.929466 12114 ts_tablet_manager.cc:1428] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260927 23:04:43.931021 12113 raft_consensus.cc:348] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.931434 12113 raft_consensus.cc:374] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:43.931571 12113 raft_consensus.cc:729] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061e7ce1e494432a56f67927fc03b66, State: Initialized, Role: FOLLOWER
I20260927 23:04:43.932102 12113 consensus_queue.cc:260] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.932548 12112 raft_consensus.cc:348] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.933020 12112 raft_consensus.cc:374] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:43.933148 12112 raft_consensus.cc:729] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cf5ee51c99463f9be1e92019449c31, State: Initialized, Role: FOLLOWER
I20260927 23:04:43.933712 12113 ts_tablet_manager.cc:1428] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Time spent starting tablet: real 0.013s user 0.001s sys 0.012s
I20260927 23:04:43.933820 12112 consensus_queue.cc:260] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.935582 12112 ts_tablet_manager.cc:1428] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 23:04:43.958961 12120 raft_consensus.cc:482] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:43.959312 12120 raft_consensus.cc:504] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.960794 12120 leader_election.cc:288] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961), 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901)
I20260927 23:04:43.967597 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.961164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33901 (local address 127.3.240.193:54441)
0927 23:04:43.961923 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:43.961938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:43.962044 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:43.962372 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:43.962376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:43.962401 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:43.962832 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.962845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.964121 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.964126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.965726 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.965737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.966878 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.966886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.967073 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.967125 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 23:04:43.967203 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 23:04:43.967282 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":474,"threads_started":1}
I20260927 23:04:43.967603 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.961302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:54441 (local address 127.3.240.195:33901)
0927 23:04:43.961523 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:43.961532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:43.962018 (+ 486us) server_negotiation.cc:407] Connection header received
0927 23:04:43.962183 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:43.962194 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:43.962259 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:43.962352 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:43.962996 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.964003 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.965890 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.966685 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.966805 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.966853 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:43.967251 (+ 398us) server_negotiation.cc:299] Negotiation successful
0927 23:04:43.967378 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:04:43.967792 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.961184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:46771 (local address 127.3.240.194:33961)
0927 23:04:43.961384 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:43.961393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:43.962596 (+ 1203us) server_negotiation.cc:407] Connection header received
0927 23:04:43.962810 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:43.962818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:43.962902 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:43.963012 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:43.963617 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.964431 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.966096 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.966854 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.967024 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.967090 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:43.967431 (+ 341us) server_negotiation.cc:299] Negotiation successful
0927 23:04:43.967534 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:04:43.967931 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:43.961236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:33961 (local address 127.3.240.193:46771)
0927 23:04:43.962485 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:43.962499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:43.962681 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:43.963044 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:43.963049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:43.963075 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:43.963442 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.963471 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.964559 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.964563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.965952 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:43.965966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:43.967100 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:43.967105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:43.967255 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:43.967285 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 23:04:43.967368 (+ 83us) client_negotiation.cc:240] Negotiation successful
0927 23:04:43.967472 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"spinlock_wait_cycles":63872,"thread_start_us":984,"threads_started":1}
I20260927 23:04:43.968884 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360f6d826ac6471298193ab67898ed5c" candidate_uuid: "c8cf5ee51c99463f9be1e92019449c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" is_pre_election: true
I20260927 23:04:43.968888 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360f6d826ac6471298193ab67898ed5c" candidate_uuid: "c8cf5ee51c99463f9be1e92019449c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0061e7ce1e494432a56f67927fc03b66" is_pre_election: true
I20260927 23:04:43.969367 11903 raft_consensus.cc:2454] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8cf5ee51c99463f9be1e92019449c31 in term 0.
I20260927 23:04:43.969365 12037 raft_consensus.cc:2454] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8cf5ee51c99463f9be1e92019449c31 in term 0.
I20260927 23:04:43.969914 11691 leader_election.cc:302] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, 8554dbb06e544027bdd8ede9b87f5cb2, c8cf5ee51c99463f9be1e92019449c31; no voters:
I20260927 23:04:43.970266 12120 raft_consensus.cc:2781] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:43.970402 12120 raft_consensus.cc:482] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:43.970507 12120 raft_consensus.cc:3037] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:43.972536 12120 raft_consensus.cc:504] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.973129 12120 leader_election.cc:288] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [CANDIDATE]: Term 1 election: Requested vote from peers 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961), 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901)
I20260927 23:04:43.973467 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360f6d826ac6471298193ab67898ed5c" candidate_uuid: "c8cf5ee51c99463f9be1e92019449c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0061e7ce1e494432a56f67927fc03b66"
I20260927 23:04:43.973572 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "360f6d826ac6471298193ab67898ed5c" candidate_uuid: "c8cf5ee51c99463f9be1e92019449c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8554dbb06e544027bdd8ede9b87f5cb2"
I20260927 23:04:43.973659 11903 raft_consensus.cc:3037] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:43.973734 12037 raft_consensus.cc:3037] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:43.976612 11903 raft_consensus.cc:2454] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8cf5ee51c99463f9be1e92019449c31 in term 1.
I20260927 23:04:43.976612 12037 raft_consensus.cc:2454] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8cf5ee51c99463f9be1e92019449c31 in term 1.
I20260927 23:04:43.976998 11692 leader_election.cc:302] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, c8cf5ee51c99463f9be1e92019449c31; no voters:
I20260927 23:04:43.977316 12120 raft_consensus.cc:2781] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:43.977684 12120 raft_consensus.cc:686] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 1 LEADER]: Becoming Leader. State: Replica: c8cf5ee51c99463f9be1e92019449c31, State: Running, Role: LEADER
I20260927 23:04:43.978104 12120 consensus_queue.cc:237] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:43.982316 11601 catalog_manager.cc:5495] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "c8cf5ee51c99463f9be1e92019449c31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } health_report { overall_health: UNKNOWN } } }
W20260927 23:04:43.986478 12088 tablet.cc:2369] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:04:43.988121 11949 tablet.cc:2369] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:04:44.001089 11810 tablet.cc:2369] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:04:44.010892 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.006035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.3.240.193:36051)
0927 23:04:44.006228 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.006234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.006250 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:04:44.006389 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.006396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.006486 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.006618 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:44.006957 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.007662 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.008890 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.009274 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.009402 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.009875 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.010343 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.010522 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.010578 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.010653 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 23:04:44.018690 12037 raft_consensus.cc:1261] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Refusing update from remote peer c8cf5ee51c99463f9be1e92019449c31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:44.018690 11903 raft_consensus.cc:1261] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Refusing update from remote peer c8cf5ee51c99463f9be1e92019449c31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:44.019711 12123 consensus_queue.cc:1035] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:44.020040 12120 consensus_queue.cc:1035] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:44.034631 12127 mvcc.cc:205] Tried to move back new op lower bound from 7334093963330576384 to 7334093963177672704. Current Snapshot: MvccSnapshot[applied={T|T < 7334093963330576384}]
I20260927 23:04:44.043892 11601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:44.044139 11601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:44.047310 11601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: Sending DeleteTablet for 3 replicas of tablet 360f6d826ac6471298193ab67898ed5c
I20260927 23:04:44.048362 11739 tablet_service.cc:1514] Processing DeleteTablet for tablet 360f6d826ac6471298193ab67898ed5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:45062
I20260927 23:04:44.048394 12017 tablet_service.cc:1514] Processing DeleteTablet for tablet 360f6d826ac6471298193ab67898ed5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39596
I20260927 23:04:44.049067 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet 360f6d826ac6471298193ab67898ed5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 23:04:44.049328 12135 tablet_replica.cc:330] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: stopping tablet replica
I20260927 23:04:44.049767 12135 raft_consensus.cc:2229] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:44.050511 12135 raft_consensus.cc:2258] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.050592 11602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:44.051208 11602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:44.052583 12135 ts_tablet_manager.cc:1905] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.058357 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0f33bbc51c4c1899ac4eca0864efc4 (DEFAULT_TABLE table=test-table [id=a385126170c94182aeb650cfa999f510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.058873 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0f33bbc51c4c1899ac4eca0864efc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.059482 12135 ts_tablet_manager.cc:1918] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:44.059618 12135 log.cc:1196] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/360f6d826ac6471298193ab67898ed5c
I20260927 23:04:44.059798 12136 tablet_replica.cc:330] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: stopping tablet replica
I20260927 23:04:44.060101 12135 ts_tablet_manager.cc:1939] T 360f6d826ac6471298193ab67898ed5c P c8cf5ee51c99463f9be1e92019449c31: Deleting consensus metadata
I20260927 23:04:44.060258 12136 raft_consensus.cc:2229] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:44.060793 12136 raft_consensus.cc:2258] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.063755 11586 catalog_manager.cc:4841] TS c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051): tablet 360f6d826ac6471298193ab67898ed5c (table test-table [id=2077c09e0001407abe3665826b617799]) successfully deleted
I20260927 23:04:44.064448 12137 tablet_replica.cc:330] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: stopping tablet replica
I20260927 23:04:44.064656 11739 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0f33bbc51c4c1899ac4eca0864efc4 (DEFAULT_TABLE table=test-table [id=a385126170c94182aeb650cfa999f510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.064841 12137 raft_consensus.cc:2229] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:44.065081 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0f33bbc51c4c1899ac4eca0864efc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.065295 12137 raft_consensus.cc:2258] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.066176 12017 tablet_service.cc:1467] Processing CreateTablet for tablet 1d0f33bbc51c4c1899ac4eca0864efc4 (DEFAULT_TABLE table=test-table [id=a385126170c94182aeb650cfa999f510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.067700 12136 ts_tablet_manager.cc:1905] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.067706 12137 ts_tablet_manager.cc:1905] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.068135 12113 tablet_bootstrap.cc:492] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: Bootstrap starting.
I20260927 23:04:44.069826 12113 tablet_bootstrap.cc:654] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.070844 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d0f33bbc51c4c1899ac4eca0864efc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.074671 12137 ts_tablet_manager.cc:1918] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:44.074762 12112 tablet_bootstrap.cc:492] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: Bootstrap starting.
I20260927 23:04:44.074918 12137 log.cc:1196] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/360f6d826ac6471298193ab67898ed5c
I20260927 23:04:44.075415 12137 ts_tablet_manager.cc:1939] T 360f6d826ac6471298193ab67898ed5c P 0061e7ce1e494432a56f67927fc03b66: Deleting consensus metadata
I20260927 23:04:44.076014 12136 ts_tablet_manager.cc:1918] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:44.076146 12136 log.cc:1196] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/360f6d826ac6471298193ab67898ed5c
I20260927 23:04:44.076350 12112 tablet_bootstrap.cc:654] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.076640 12136 ts_tablet_manager.cc:1939] T 360f6d826ac6471298193ab67898ed5c P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting consensus metadata
I20260927 23:04:44.077183 11589 catalog_manager.cc:4841] TS 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961): tablet 360f6d826ac6471298193ab67898ed5c (table test-table [id=2077c09e0001407abe3665826b617799]) successfully deleted
I20260927 23:04:44.077843 12114 tablet_bootstrap.cc:492] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: Bootstrap starting.
I20260927 23:04:44.079160 12114 tablet_bootstrap.cc:654] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.080935 12114 tablet_bootstrap.cc:492] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: No bootstrap required, opened a new log
I20260927 23:04:44.081099 12114 ts_tablet_manager.cc:1397] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260927 23:04:44.081830 12114 raft_consensus.cc:348] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.082161 12114 raft_consensus.cc:374] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.082273 12114 raft_consensus.cc:729] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8554dbb06e544027bdd8ede9b87f5cb2, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.082559 12114 consensus_queue.cc:260] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.083228 12114 ts_tablet_manager.cc:1428] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 23:04:44.087754 11588 catalog_manager.cc:4841] TS 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901): tablet 360f6d826ac6471298193ab67898ed5c (table test-table [id=2077c09e0001407abe3665826b617799]) successfully deleted
I20260927 23:04:44.090950 12112 tablet_bootstrap.cc:492] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: No bootstrap required, opened a new log
I20260927 23:04:44.091150 12112 ts_tablet_manager.cc:1397] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20260927 23:04:44.091892 12113 tablet_bootstrap.cc:492] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: No bootstrap required, opened a new log
I20260927 23:04:44.092048 12113 ts_tablet_manager.cc:1397] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.015s
I20260927 23:04:44.092355 12112 raft_consensus.cc:348] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.092854 12113 raft_consensus.cc:348] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.092995 12112 raft_consensus.cc:374] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.093072 12113 raft_consensus.cc:374] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.093103 12112 raft_consensus.cc:729] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cf5ee51c99463f9be1e92019449c31, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.093165 12113 raft_consensus.cc:729] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061e7ce1e494432a56f67927fc03b66, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.093391 12113 consensus_queue.cc:260] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.093393 12112 consensus_queue.cc:260] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.094031 12113 ts_tablet_manager.cc:1428] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260927 23:04:44.094041 12112 ts_tablet_manager.cc:1428] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260927 23:04:44.248018 12119 raft_consensus.cc:482] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:44.248229 12119 raft_consensus.cc:504] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.250083 12119 leader_election.cc:288] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051), 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901)
I20260927 23:04:44.265364 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.250589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:60043 (local address 127.3.240.193:36051)
0927 23:04:44.250813 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.250822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.259199 (+ 8377us) server_negotiation.cc:407] Connection header received
0927 23:04:44.259487 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.259494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.259582 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.259687 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:44.260304 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.261159 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.263409 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.264422 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.264553 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.264614 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.264951 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.265079 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:04:44.265360 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.250252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36051 (local address 127.3.240.194:60043)
0927 23:04:44.259093 (+ 8841us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:44.259114 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:44.259316 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:44.259733 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:44.259739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:44.259768 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:44.260142 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.260153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.261275 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.261282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.263251 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.263264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.264613 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.264620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.264804 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.264824 (+ 20us) client_negotiation.cc:769] Sending connection context
0927 23:04:44.264928 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 23:04:44.265055 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8683,"thread_start_us":8918,"threads_started":1}
I20260927 23:04:44.265458 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.251710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:58887 (local address 127.3.240.195:33901)
0927 23:04:44.251860 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.251865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.259578 (+ 7713us) server_negotiation.cc:407] Connection header received
0927 23:04:44.259827 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.259846 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.259913 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.259958 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:44.260705 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.261705 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.263833 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.264535 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.264679 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.264745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.265104 (+ 359us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.265225 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:04:44.266026 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.251260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33901 (local address 127.3.240.194:58887)
0927 23:04:44.259487 (+ 8227us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:44.259503 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:44.259674 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:44.260118 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:44.260120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:44.260149 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:44.260519 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.260531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.261898 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.261903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.263696 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.263704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.264763 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.264768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.264930 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.264956 (+ 26us) client_negotiation.cc:769] Sending connection context
0927 23:04:44.265598 (+ 642us) client_negotiation.cc:240] Negotiation successful
0927 23:04:44.265713 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8069,"mutex_wait_us":1,"thread_start_us":7521,"threads_started":1}
I20260927 23:04:44.266254 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0f33bbc51c4c1899ac4eca0864efc4" candidate_uuid: "0061e7ce1e494432a56f67927fc03b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8cf5ee51c99463f9be1e92019449c31" is_pre_election: true
I20260927 23:04:44.266714 11759 raft_consensus.cc:2454] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0061e7ce1e494432a56f67927fc03b66 in term 0.
I20260927 23:04:44.266717 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0f33bbc51c4c1899ac4eca0864efc4" candidate_uuid: "0061e7ce1e494432a56f67927fc03b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" is_pre_election: true
I20260927 23:04:44.266937 12037 raft_consensus.cc:2454] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0061e7ce1e494432a56f67927fc03b66 in term 0.
I20260927 23:04:44.267335 11829 leader_election.cc:302] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, c8cf5ee51c99463f9be1e92019449c31; no voters:
I20260927 23:04:44.267673 12119 raft_consensus.cc:2781] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:44.267788 12119 raft_consensus.cc:482] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:44.267885 12119 raft_consensus.cc:3037] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.269873 12119 raft_consensus.cc:504] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.270450 12119 leader_election.cc:288] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [CANDIDATE]: Term 1 election: Requested vote from peers c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051), 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901)
I20260927 23:04:44.270816 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0f33bbc51c4c1899ac4eca0864efc4" candidate_uuid: "0061e7ce1e494432a56f67927fc03b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8cf5ee51c99463f9be1e92019449c31"
I20260927 23:04:44.270895 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d0f33bbc51c4c1899ac4eca0864efc4" candidate_uuid: "0061e7ce1e494432a56f67927fc03b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8554dbb06e544027bdd8ede9b87f5cb2"
I20260927 23:04:44.271040 11759 raft_consensus.cc:3037] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.271081 12037 raft_consensus.cc:3037] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.273864 12037 raft_consensus.cc:2454] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0061e7ce1e494432a56f67927fc03b66 in term 1.
I20260927 23:04:44.274151 11759 raft_consensus.cc:2454] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0061e7ce1e494432a56f67927fc03b66 in term 1.
I20260927 23:04:44.274224 11831 leader_election.cc:302] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, 8554dbb06e544027bdd8ede9b87f5cb2; no voters:
I20260927 23:04:44.274547 12119 raft_consensus.cc:2781] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:44.275141 12119 raft_consensus.cc:686] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 1 LEADER]: Becoming Leader. State: Replica: 0061e7ce1e494432a56f67927fc03b66, State: Running, Role: LEADER
I20260927 23:04:44.275509 12119 consensus_queue.cc:237] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.288081 11602 catalog_manager.cc:5495] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "0061e7ce1e494432a56f67927fc03b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } health_report { overall_health: UNKNOWN } } }
Sep 27 23:04:44 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550281, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:44.347718 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.333457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39761 (local address 127.0.0.1:51204)
0927 23:04:44.334051 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:44.334070 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:44.334188 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:44.334529 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:44.334532 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:44.335259 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:44.335522 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.335531 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.336872 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.336875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.337749 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.337759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.338690 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.338694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.338807 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.339471 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:44.339488 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:44.341696 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:44.345291 (+ 3595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:44.345304 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:44.345310 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:44.345697 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:44.346099 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:44.346102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:44.346104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:44.346278 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:44.346897 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:44.346902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:44.347093 (+ 191us) client_negotiation.cc:769] Sending connection context
0927 23:04:44.347223 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 23:04:44.347399 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":338,"threads_started":1}
I20260927 23:04:44.348217 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.333435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51204 (local address 127.3.240.254:39761)
0927 23:04:44.333644 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.333652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.334177 (+ 525us) server_negotiation.cc:407] Connection header received
0927 23:04:44.334297 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.334303 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.334378 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.334492 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:44.335668 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.336762 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.337900 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.338522 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.338645 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.341896 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:44.341904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:44.341908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:44.341946 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:44.345113 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:44.345815 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:44.345822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:44.345825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:44.345985 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:44.346416 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:44.346422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:44.346427 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:44.346763 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:44.346936 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.347355 (+ 419us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.347598 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 23:04:44.353011 11602 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:51204:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260927 23:04:44.354246 11602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:44.361052 11739 tablet_service.cc:1467] Processing CreateTablet for tablet 29e83c963fcb43ceba5421495d1681c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65d74c6122e04c23a61279ffb68f85a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:04:44.361243 12017 tablet_service.cc:1467] Processing CreateTablet for tablet 29e83c963fcb43ceba5421495d1681c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65d74c6122e04c23a61279ffb68f85a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:04:44.361580 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e83c963fcb43ceba5421495d1681c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.361774 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e83c963fcb43ceba5421495d1681c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.366470 12112 tablet_bootstrap.cc:492] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: Bootstrap starting.
I20260927 23:04:44.366932 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 29e83c963fcb43ceba5421495d1681c6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=65d74c6122e04c23a61279ffb68f85a2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260927 23:04:44.367473 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e83c963fcb43ceba5421495d1681c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.368198 12112 tablet_bootstrap.cc:654] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.368561 12114 tablet_bootstrap.cc:492] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: Bootstrap starting.
I20260927 23:04:44.370113 12114 tablet_bootstrap.cc:654] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.376366 12113 tablet_bootstrap.cc:492] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: Bootstrap starting.
I20260927 23:04:44.378093 12113 tablet_bootstrap.cc:654] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.382136 12114 tablet_bootstrap.cc:492] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: No bootstrap required, opened a new log
I20260927 23:04:44.382225 12113 tablet_bootstrap.cc:492] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: No bootstrap required, opened a new log
I20260927 23:04:44.382375 12113 ts_tablet_manager.cc:1397] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260927 23:04:44.382380 12114 ts_tablet_manager.cc:1397] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260927 23:04:44.382755 12112 tablet_bootstrap.cc:492] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: No bootstrap required, opened a new log
I20260927 23:04:44.383071 12112 ts_tablet_manager.cc:1397] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20260927 23:04:44.383119 12113 raft_consensus.cc:348] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.383359 12113 raft_consensus.cc:374] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.383239 12114 raft_consensus.cc:348] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.383432 12113 raft_consensus.cc:729] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061e7ce1e494432a56f67927fc03b66, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.383461 12114 raft_consensus.cc:374] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.383531 12114 raft_consensus.cc:729] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8554dbb06e544027bdd8ede9b87f5cb2, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.383641 12113 consensus_queue.cc:260] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.383786 12114 consensus_queue.cc:260] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.383852 12112 raft_consensus.cc:348] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.384073 12112 raft_consensus.cc:374] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.384145 12112 raft_consensus.cc:729] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cf5ee51c99463f9be1e92019449c31, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.384195 12113 ts_tablet_manager.cc:1428] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260927 23:04:44.384215 12118 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.384531 12118 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:04:44.384425 12112 consensus_queue.cc:260] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.384629 12119 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.384884 12119 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:04:44.384801 12120 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.385035 12120 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:04:44.385512 12112 ts_tablet_manager.cc:1428] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260927 23:04:44.385766 12114 ts_tablet_manager.cc:1428] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260927 23:04:44.412185 12118 raft_consensus.cc:482] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:44.412380 12118 raft_consensus.cc:504] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.413977 12118 leader_election.cc:288] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051), 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961)
I20260927 23:04:44.421947 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.414755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:56457 (local address 127.3.240.194:33961)
0927 23:04:44.414935 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.414941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.415820 (+ 879us) server_negotiation.cc:407] Connection header received
0927 23:04:44.416054 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.416059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.416131 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.416213 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:44.417069 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.418070 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.419905 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.420933 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.421088 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.421163 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.421524 (+ 361us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.421650 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260927 23:04:44.422060 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.414533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:33961 (local address 127.3.240.195:56457)
0927 23:04:44.415734 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:44.415752 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:44.415889 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:44.416403 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:44.416408 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:44.416429 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:44.416853 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.416867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.418183 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.418189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.419759 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.419769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.421155 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.421162 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.421344 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.421378 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 23:04:44.421485 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 23:04:44.421619 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":24,"thread_start_us":850,"threads_started":1}
I20260927 23:04:44.422715 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e83c963fcb43ceba5421495d1681c6" candidate_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0061e7ce1e494432a56f67927fc03b66" is_pre_election: true
I20260927 23:04:44.422981 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.414555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47913 (local address 127.3.240.193:36051)
0927 23:04:44.414740 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.414747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.415547 (+ 800us) server_negotiation.cc:407] Connection header received
0927 23:04:44.415807 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.415812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.415890 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.415985 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:04:44.416646 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.417806 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.420401 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.421661 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.421839 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.421899 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.422557 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.422700 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 23:04:44.422966 11903 raft_consensus.cc:2454] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8554dbb06e544027bdd8ede9b87f5cb2 in term 0.
I20260927 23:04:44.423029 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.414181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36051 (local address 127.3.240.195:47913)
0927 23:04:44.415424 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:44.415452 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:44.415651 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:44.415999 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:44.416003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:44.416024 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:04:44.416421 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.416433 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.417967 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.417973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.420222 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.420237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.422077 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.422084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.422397 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.422437 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 23:04:44.422534 (+ 97us) client_negotiation.cc:240] Negotiation successful
0927 23:04:44.422662 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":25,"thread_start_us":879,"threads_started":1}
I20260927 23:04:44.423552 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e83c963fcb43ceba5421495d1681c6" candidate_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8cf5ee51c99463f9be1e92019449c31" is_pre_election: true
I20260927 23:04:44.423681 11971 leader_election.cc:302] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, 8554dbb06e544027bdd8ede9b87f5cb2; no voters:
I20260927 23:04:44.423810 11759 raft_consensus.cc:2454] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8554dbb06e544027bdd8ede9b87f5cb2 in term 0.
I20260927 23:04:44.424271 12118 raft_consensus.cc:2781] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:44.424393 12118 raft_consensus.cc:482] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:44.424484 12118 raft_consensus.cc:3037] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.426538 12118 raft_consensus.cc:504] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.427099 12118 leader_election.cc:288] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [CANDIDATE]: Term 1 election: Requested vote from peers c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051), 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961)
I20260927 23:04:44.427556 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e83c963fcb43ceba5421495d1681c6" candidate_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8cf5ee51c99463f9be1e92019449c31"
I20260927 23:04:44.427608 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e83c963fcb43ceba5421495d1681c6" candidate_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0061e7ce1e494432a56f67927fc03b66"
I20260927 23:04:44.427798 11903 raft_consensus.cc:3037] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.427778 11759 raft_consensus.cc:3037] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.430639 11903 raft_consensus.cc:2454] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8554dbb06e544027bdd8ede9b87f5cb2 in term 1.
I20260927 23:04:44.430838 11759 raft_consensus.cc:2454] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8554dbb06e544027bdd8ede9b87f5cb2 in term 1.
I20260927 23:04:44.431082 11971 leader_election.cc:302] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, 8554dbb06e544027bdd8ede9b87f5cb2; no voters:
I20260927 23:04:44.431406 12118 raft_consensus.cc:2781] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:44.431890 12118 raft_consensus.cc:686] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 8554dbb06e544027bdd8ede9b87f5cb2, State: Running, Role: LEADER
I20260927 23:04:44.432485 12118 consensus_queue.cc:237] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.434279 12156 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: TxnStatusTablet state changed. Reason: New leader 8554dbb06e544027bdd8ede9b87f5cb2. Latest consensus state: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.434656 12156 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:04:44.435322 12158 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260927 23:04:44.435578 12158 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260927 23:04:44.436039 11600 catalog_manager.cc:5495] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195). New cstate: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } health_report { overall_health: HEALTHY } } }
I20260927 23:04:44.468214 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.460975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39604 (local address 127.3.240.195:33901)
0927 23:04:44.461665 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.461670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.461687 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:04:44.461767 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.461771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.461894 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.462018 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:44.462424 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.463664 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.465545 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.466206 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.466342 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.466842 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.467560 (+ 718us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.467715 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.467834 (+ 119us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.467926 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555}
I20260927 23:04:44.468238 12150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.460944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33901 (local address 127.0.0.1:39604)
0927 23:04:44.461257 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:44.461275 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:44.461395 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:44.462072 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:44.462077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:44.462090 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:04:44.462278 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.462290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.463818 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.463824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.465375 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.465389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.466378 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.466384 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.466580 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.466649 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.467679 (+ 1030us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.467708 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 23:04:44.467798 (+ 90us) client_negotiation.cc:240] Negotiation successful
0927 23:04:44.467945 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260927 23:04:44.472404 11903 raft_consensus.cc:1261] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Refusing update from remote peer 8554dbb06e544027bdd8ede9b87f5cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:44.472775 11759 raft_consensus.cc:1261] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Refusing update from remote peer 8554dbb06e544027bdd8ede9b87f5cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:44.473361 12118 consensus_queue.cc:1035] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:44.473770 12156 consensus_queue.cc:1035] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:44.485502 12119 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: TxnStatusTablet state changed. Reason: New leader 8554dbb06e544027bdd8ede9b87f5cb2. Latest consensus state: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.485780 12119 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:04:44.488906 12156 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.489142 12156 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:04:44.490111 12128 mvcc.cc:205] Tried to move back new op lower bound from 7334093965192187904 to 7334093965038448640. Current Snapshot: MvccSnapshot[applied={T|T < 7334093965192187904}]
I20260927 23:04:44.490362 12119 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.490597 12119 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P 0061e7ce1e494432a56f67927fc03b66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:04:44.493304 12120 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: TxnStatusTablet state changed. Reason: New leader 8554dbb06e544027bdd8ede9b87f5cb2. Latest consensus state: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.493556 12120 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:04:44.493396 12118 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.493631 12118 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P 8554dbb06e544027bdd8ede9b87f5cb2: This TxnStatusTablet replica's current role is: LEADER
I20260927 23:04:44.495041 12126 mvcc.cc:205] Tried to move back new op lower bound from 7334093965192187904 to 7334093965038448640. Current Snapshot: MvccSnapshot[applied={T|T < 7334093965192187904}]
I20260927 23:04:44.495985 12120 tablet_replica.cc:439] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } } }
I20260927 23:04:44.496229 12120 tablet_replica.cc:442] T 29e83c963fcb43ceba5421495d1681c6 P c8cf5ee51c99463f9be1e92019449c31: This TxnStatusTablet replica's current role is: FOLLOWER
I20260927 23:04:44.506466 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.498576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.3.240.194:33961)
0927 23:04:44.498729 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.498736 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.498756 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:04:44.498849 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.498853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.498966 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.499133 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:44.499669 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.500794 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.503596 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.504223 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.504353 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.504924 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.505618 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.505757 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.505887 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.505971 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 23:04:44.523861 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.518201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:34123 (local address 127.3.240.195:33901)
0927 23:04:44.518398 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.518405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.519022 (+ 617us) server_negotiation.cc:407] Connection header received
0927 23:04:44.519210 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.519216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.519307 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.519421 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:44.519863 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.520573 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.521987 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.522463 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.522622 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.523072 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.523175 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.523278 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.523442 (+ 164us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.523521 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 23:04:44.523919 12165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.518178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33901 (local address 127.3.240.194:34123)
0927 23:04:44.518882 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:44.518899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:44.519084 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:44.519430 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:44.519433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:44.519442 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:04:44.519721 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.519727 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.520730 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.520736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.521824 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.521836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.522665 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.522673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.522794 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.522843 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.523288 (+ 445us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.523315 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 23:04:44.523417 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 23:04:44.523545 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":378,"threads_started":1}
I20260927 23:04:44.533067 11759 raft_consensus.cc:1261] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Refusing update from remote peer 0061e7ce1e494432a56f67927fc03b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:44.533105 12037 raft_consensus.cc:1261] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Refusing update from remote peer 0061e7ce1e494432a56f67927fc03b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:44.533723 12143 consensus_queue.cc:1035] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:44.534083 12119 consensus_queue.cc:1035] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:44.579712 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.572792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:56929 (local address 127.3.240.194:33961)
0927 23:04:44.572989 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.572996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.574023 (+ 1027us) server_negotiation.cc:407] Connection header received
0927 23:04:44.574268 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.574273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.574343 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.574448 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:44.575232 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.575964 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.577828 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.578414 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.578544 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.579147 (+ 603us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.579236 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.579303 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.579446 (+ 143us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.579501 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 23:04:44.579999 12170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.572845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:33961 (local address 127.3.240.195:56929)
0927 23:04:44.573908 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:44.573927 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:44.574138 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:44.574585 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:44.574589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:44.574604 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:04:44.575052 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.575065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.576102 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.576105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.577490 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:44.577503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.578700 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.578707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:44.578896 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.578962 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.579334 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.579344 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 23:04:44.579436 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 23:04:44.579568 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"thread_start_us":664,"threads_started":1}
I20260927 23:04:44.617141 11600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:44.617398 11600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:44.620645 11600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: Sending DeleteTablet for 3 replicas of tablet 1d0f33bbc51c4c1899ac4eca0864efc4
I20260927 23:04:44.621351 11739 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0f33bbc51c4c1899ac4eca0864efc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:45062
I20260927 23:04:44.621542 12017 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0f33bbc51c4c1899ac4eca0864efc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39596
I20260927 23:04:44.622411 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d0f33bbc51c4c1899ac4eca0864efc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 23:04:44.627384 12175 tablet_replica.cc:330] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: stopping tablet replica
I20260927 23:04:44.627638 12175 raft_consensus.cc:2229] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:44.628279 12175 raft_consensus.cc:2258] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.632457 12175 ts_tablet_manager.cc:1905] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.636020 12174 tablet_replica.cc:330] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: stopping tablet replica
I20260927 23:04:44.636317 12174 raft_consensus.cc:2229] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:44.636643 12174 raft_consensus.cc:2258] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.637827 12174 ts_tablet_manager.cc:1905] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.640015 12175 ts_tablet_manager.cc:1918] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:44.645043 12174 ts_tablet_manager.cc:1918] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:44.645262 12174 log.cc:1196] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/1d0f33bbc51c4c1899ac4eca0864efc4
I20260927 23:04:44.645848 12174 ts_tablet_manager.cc:1939] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting consensus metadata
I20260927 23:04:44.645879 12173 tablet_replica.cc:330] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: stopping tablet replica
I20260927 23:04:44.646040 12175 log.cc:1196] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/1d0f33bbc51c4c1899ac4eca0864efc4
I20260927 23:04:44.646162 12173 raft_consensus.cc:2229] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:44.646412 12173 raft_consensus.cc:2258] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.647137 11588 catalog_manager.cc:4841] TS 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901): tablet 1d0f33bbc51c4c1899ac4eca0864efc4 (table test-table [id=a385126170c94182aeb650cfa999f510]) successfully deleted
I20260927 23:04:44.647415 12173 ts_tablet_manager.cc:1905] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.654078 12173 ts_tablet_manager.cc:1918] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:44.654198 12173 log.cc:1196] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/1d0f33bbc51c4c1899ac4eca0864efc4
I20260927 23:04:44.654533 12175 ts_tablet_manager.cc:1939] T 1d0f33bbc51c4c1899ac4eca0864efc4 P 0061e7ce1e494432a56f67927fc03b66: Deleting consensus metadata
I20260927 23:04:44.654666 12173 ts_tablet_manager.cc:1939] T 1d0f33bbc51c4c1899ac4eca0864efc4 P c8cf5ee51c99463f9be1e92019449c31: Deleting consensus metadata
I20260927 23:04:44.655716 11586 catalog_manager.cc:4841] TS c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051): tablet 1d0f33bbc51c4c1899ac4eca0864efc4 (table test-table [id=a385126170c94182aeb650cfa999f510]) successfully deleted
I20260927 23:04:44.655712 11589 catalog_manager.cc:4841] TS 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961): tablet 1d0f33bbc51c4c1899ac4eca0864efc4 (table test-table [id=a385126170c94182aeb650cfa999f510]) successfully deleted
Sep 27 23:04:44 dist-test-slave-lmfp krb5kdc[11557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 27 23:04:44 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:44 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 23:04:44.709892 11697 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:44.696589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51210 (local address 127.3.240.254:39761)
0927 23:04:44.696765 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.696771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.697295 (+ 524us) server_negotiation.cc:407] Connection header received
0927 23:04:44.697565 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.697571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.697653 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.697763 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:44.699382 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.700499 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.702401 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.703172 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.703330 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.709630 (+ 6300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51210: BlockingRecv error: recv got EOF from 127.0.0.1:51210 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 23:04:44 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260927 23:04:44.731722 11697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.713614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.3.240.254:39761)
0927 23:04:44.713757 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.713762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.715426 (+ 1664us) server_negotiation.cc:407] Connection header received
0927 23:04:44.715658 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.715663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.715735 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.715836 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:44.717201 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.718198 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.719972 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.720553 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.720703 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.725135 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:44.725142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:44.725146 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:44.725183 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:44.728513 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:44.729261 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:44.729266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:44.729270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:44.729418 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:44.729852 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:44.729857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:44.729861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:44.730187 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:44.730396 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.731158 (+ 762us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.731380 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:04:44.735615 11600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:44.736304 11600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:44.743587 11739 tablet_service.cc:1467] Processing CreateTablet for tablet 6499094453d54d05b97d50f2e1717509 (DEFAULT_TABLE table=test-table [id=a9e2fe2c52394f81a4df41ea289db6ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.743817 12017 tablet_service.cc:1467] Processing CreateTablet for tablet 6499094453d54d05b97d50f2e1717509 (DEFAULT_TABLE table=test-table [id=a9e2fe2c52394f81a4df41ea289db6ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.744009 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6499094453d54d05b97d50f2e1717509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.744252 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6499094453d54d05b97d50f2e1717509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.751659 12114 tablet_bootstrap.cc:492] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: Bootstrap starting.
I20260927 23:04:44.751889 11877 tablet_service.cc:1467] Processing CreateTablet for tablet 6499094453d54d05b97d50f2e1717509 (DEFAULT_TABLE table=test-table [id=a9e2fe2c52394f81a4df41ea289db6ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.752321 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6499094453d54d05b97d50f2e1717509. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.753346 12114 tablet_bootstrap.cc:654] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.772853 12112 tablet_bootstrap.cc:492] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: Bootstrap starting.
I20260927 23:04:44.774528 12112 tablet_bootstrap.cc:654] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.777813 12113 tablet_bootstrap.cc:492] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: Bootstrap starting.
I20260927 23:04:44.779539 12113 tablet_bootstrap.cc:654] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.791842 12113 tablet_bootstrap.cc:492] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: No bootstrap required, opened a new log
I20260927 23:04:44.792057 12113 ts_tablet_manager.cc:1397] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20260927 23:04:44.793115 12113 raft_consensus.cc:348] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.793462 12113 raft_consensus.cc:374] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.793547 12113 raft_consensus.cc:729] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061e7ce1e494432a56f67927fc03b66, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.793846 12113 consensus_queue.cc:260] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.794464 12113 ts_tablet_manager.cc:1428] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260927 23:04:44.798266 12112 tablet_bootstrap.cc:492] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: No bootstrap required, opened a new log
I20260927 23:04:44.798588 12112 ts_tablet_manager.cc:1397] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.001s
I20260927 23:04:44.799501 12112 raft_consensus.cc:348] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.799842 12112 raft_consensus.cc:374] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.799979 12112 raft_consensus.cc:729] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cf5ee51c99463f9be1e92019449c31, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.800280 12112 consensus_queue.cc:260] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.803053 12112 ts_tablet_manager.cc:1428] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 23:04:44.813232 12114 tablet_bootstrap.cc:492] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: No bootstrap required, opened a new log
I20260927 23:04:44.813444 12114 ts_tablet_manager.cc:1397] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent bootstrapping tablet: real 0.062s user 0.012s sys 0.035s
I20260927 23:04:44.814265 12114 raft_consensus.cc:348] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.814592 12114 raft_consensus.cc:374] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.814738 12114 raft_consensus.cc:729] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8554dbb06e544027bdd8ede9b87f5cb2, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.815088 12114 consensus_queue.cc:260] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.817763 12114 ts_tablet_manager.cc:1428] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260927 23:04:44.854513 12168 raft_consensus.cc:482] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:44.854703 12168 raft_consensus.cc:504] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.855381 12168 leader_election.cc:288] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051), 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901)
I20260927 23:04:44.855825 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6499094453d54d05b97d50f2e1717509" candidate_uuid: "0061e7ce1e494432a56f67927fc03b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8cf5ee51c99463f9be1e92019449c31" is_pre_election: true
I20260927 23:04:44.856055 11759 raft_consensus.cc:2454] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0061e7ce1e494432a56f67927fc03b66 in term 0.
I20260927 23:04:44.856019 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6499094453d54d05b97d50f2e1717509" candidate_uuid: "0061e7ce1e494432a56f67927fc03b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" is_pre_election: true
I20260927 23:04:44.856232 12037 raft_consensus.cc:2454] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0061e7ce1e494432a56f67927fc03b66 in term 0.
I20260927 23:04:44.856523 11829 leader_election.cc:302] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, c8cf5ee51c99463f9be1e92019449c31; no voters:
I20260927 23:04:44.856891 12168 raft_consensus.cc:2781] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:44.857117 12168 raft_consensus.cc:482] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:44.857254 12168 raft_consensus.cc:3037] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.860986 12168 raft_consensus.cc:504] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.861984 12168 leader_election.cc:288] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [CANDIDATE]: Term 1 election: Requested vote from peers c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051), 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901)
I20260927 23:04:44.862046 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6499094453d54d05b97d50f2e1717509" candidate_uuid: "0061e7ce1e494432a56f67927fc03b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8cf5ee51c99463f9be1e92019449c31"
I20260927 23:04:44.862234 11759 raft_consensus.cc:3037] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.862396 12037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6499094453d54d05b97d50f2e1717509" candidate_uuid: "0061e7ce1e494432a56f67927fc03b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8554dbb06e544027bdd8ede9b87f5cb2"
I20260927 23:04:44.862599 12037 raft_consensus.cc:3037] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:44.865098 11759 raft_consensus.cc:2454] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0061e7ce1e494432a56f67927fc03b66 in term 1.
I20260927 23:04:44.865516 11829 leader_election.cc:302] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, c8cf5ee51c99463f9be1e92019449c31; no voters:
I20260927 23:04:44.865711 12037 raft_consensus.cc:2454] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0061e7ce1e494432a56f67927fc03b66 in term 1.
I20260927 23:04:44.866081 12168 raft_consensus.cc:2781] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:44.866277 12168 raft_consensus.cc:686] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 1 LEADER]: Becoming Leader. State: Replica: 0061e7ce1e494432a56f67927fc03b66, State: Running, Role: LEADER
I20260927 23:04:44.866652 12168 consensus_queue.cc:237] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.869390 11601 catalog_manager.cc:5495] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "0061e7ce1e494432a56f67927fc03b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } health_report { overall_health: UNKNOWN } } }
I20260927 23:04:44.897085 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:44.890534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41886 (local address 127.3.240.194:33961)
0927 23:04:44.890712 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:44.890718 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:44.890917 (+ 199us) server_negotiation.cc:407] Connection header received
0927 23:04:44.891133 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:44.891140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:44.891229 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:44.891327 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:44.891812 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.892948 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.894847 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:44.895599 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:44.895731 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:44.896322 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:44.896454 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:44.896547 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:44.896705 (+ 158us) server_negotiation.cc:299] Negotiation successful
0927 23:04:44.896793 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:04:44.899825 12037 raft_consensus.cc:1261] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Refusing update from remote peer 0061e7ce1e494432a56f67927fc03b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:44.900386 11759 raft_consensus.cc:1261] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Refusing update from remote peer 0061e7ce1e494432a56f67927fc03b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:44.900413 12143 consensus_queue.cc:1035] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:44.901369 12143 consensus_queue.cc:1035] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:44.921797 11600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:44.922070 11600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:44.927459 11600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: Sending DeleteTablet for 3 replicas of tablet 6499094453d54d05b97d50f2e1717509
I20260927 23:04:44.928442 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 6499094453d54d05b97d50f2e1717509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 23:04:44.928699 12175 tablet_replica.cc:330] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: stopping tablet replica
I20260927 23:04:44.928871 12175 raft_consensus.cc:2229] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:44.929037 12016 tablet_service.cc:1514] Processing DeleteTablet for tablet 6499094453d54d05b97d50f2e1717509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39596
I20260927 23:04:44.929037 11739 tablet_service.cc:1514] Processing DeleteTablet for tablet 6499094453d54d05b97d50f2e1717509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:45062
I20260927 23:04:44.929116 12175 raft_consensus.cc:2258] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.929411 12017 tablet_service.cc:1514] Processing DeleteTablet for tablet 6499094453d54d05b97d50f2e1717509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39596
I20260927 23:04:44.929623 12174 tablet_replica.cc:330] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: stopping tablet replica
I20260927 23:04:44.929791 12174 raft_consensus.cc:2229] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:44.929983 12174 raft_consensus.cc:2258] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.930204 12173 tablet_replica.cc:330] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: stopping tablet replica
I20260927 23:04:44.930373 12173 raft_consensus.cc:2229] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:44.930573 12173 raft_consensus.cc:2258] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:44.930727 12175 ts_tablet_manager.cc:1905] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.931625 11602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:04:44.932248 11602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:04:44.936321 12173 ts_tablet_manager.cc:1905] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.936522 12174 ts_tablet_manager.cc:1905] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:44.940306 11739 tablet_service.cc:1467] Processing CreateTablet for tablet f4ceb94a107b4b08b152b385336f2f0c (DEFAULT_TABLE table=test-table [id=9334ae04147645f6859927c4f4d8f02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.940685 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ceb94a107b4b08b152b385336f2f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.942866 12174 ts_tablet_manager.cc:1918] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:44.943053 12174 log.cc:1196] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/6499094453d54d05b97d50f2e1717509
I20260927 23:04:44.943490 12174 ts_tablet_manager.cc:1939] T 6499094453d54d05b97d50f2e1717509 P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting consensus metadata
I20260927 23:04:44.944985 11877 tablet_service.cc:1514] Processing DeleteTablet for tablet 6499094453d54d05b97d50f2e1717509 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:44 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 23:04:44.945204 11878 tablet_service.cc:1467] Processing CreateTablet for tablet f4ceb94a107b4b08b152b385336f2f0c (DEFAULT_TABLE table=test-table [id=9334ae04147645f6859927c4f4d8f02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.945976 11588 catalog_manager.cc:4841] TS 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901): tablet 6499094453d54d05b97d50f2e1717509 (table test-table [id=a9e2fe2c52394f81a4df41ea289db6ae]) successfully deleted
W20260927 23:04:44.946331 11588 catalog_manager.cc:4808] TS 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901): delete failed for tablet 6499094453d54d05b97d50f2e1717509 because the tablet was not found. No further retry: Not found: Tablet not found: 6499094453d54d05b97d50f2e1717509
I20260927 23:04:44.946388 12017 tablet_service.cc:1467] Processing CreateTablet for tablet f4ceb94a107b4b08b152b385336f2f0c (DEFAULT_TABLE table=test-table [id=9334ae04147645f6859927c4f4d8f02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:04:44.946753 12017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ceb94a107b4b08b152b385336f2f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:44.947031 12175 ts_tablet_manager.cc:1918] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:44.947170 12175 log.cc:1196] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/6499094453d54d05b97d50f2e1717509
I20260927 23:04:44.947613 12175 ts_tablet_manager.cc:1939] T 6499094453d54d05b97d50f2e1717509 P 0061e7ce1e494432a56f67927fc03b66: Deleting consensus metadata
I20260927 23:04:44.948560 11589 catalog_manager.cc:4841] TS 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961): tablet 6499094453d54d05b97d50f2e1717509 (table test-table [id=a9e2fe2c52394f81a4df41ea289db6ae]) successfully deleted
I20260927 23:04:44.951263 11878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ceb94a107b4b08b152b385336f2f0c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260927 23:04:44.955708 11589 catalog_manager.cc:4808] TS 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961): delete failed for tablet 6499094453d54d05b97d50f2e1717509 because the tablet was not found. No further retry: Not found: Tablet not found: 6499094453d54d05b97d50f2e1717509
I20260927 23:04:44.957940 12113 tablet_bootstrap.cc:492] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: Bootstrap starting.
I20260927 23:04:44.958406 12173 ts_tablet_manager.cc:1918] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:04:44.958611 12173 log.cc:1196] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/6499094453d54d05b97d50f2e1717509
I20260927 23:04:44.959138 12173 ts_tablet_manager.cc:1939] T 6499094453d54d05b97d50f2e1717509 P c8cf5ee51c99463f9be1e92019449c31: Deleting consensus metadata
I20260927 23:04:44.959663 12113 tablet_bootstrap.cc:654] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.960140 12114 tablet_bootstrap.cc:492] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: Bootstrap starting.
I20260927 23:04:44.961660 12114 tablet_bootstrap.cc:654] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.966331 11586 catalog_manager.cc:4841] TS c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051): tablet 6499094453d54d05b97d50f2e1717509 (table test-table [id=a9e2fe2c52394f81a4df41ea289db6ae]) successfully deleted
I20260927 23:04:44.971396 12114 tablet_bootstrap.cc:492] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: No bootstrap required, opened a new log
I20260927 23:04:44.971571 12112 tablet_bootstrap.cc:492] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: Bootstrap starting.
I20260927 23:04:44.971627 12114 ts_tablet_manager.cc:1397] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260927 23:04:44.972470 12114 raft_consensus.cc:348] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.972891 12114 raft_consensus.cc:374] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.973018 12114 raft_consensus.cc:729] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8554dbb06e544027bdd8ede9b87f5cb2, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.973399 12114 consensus_queue.cc:260] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.973732 12112 tablet_bootstrap.cc:654] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:44.974922 12114 ts_tablet_manager.cc:1428] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260927 23:04:44.976666 12113 tablet_bootstrap.cc:492] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: No bootstrap required, opened a new log
I20260927 23:04:44.976861 12113 ts_tablet_manager.cc:1397] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.002s
I20260927 23:04:44.977581 12113 raft_consensus.cc:348] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.977878 12112 tablet_bootstrap.cc:492] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: No bootstrap required, opened a new log
I20260927 23:04:44.977869 12113 raft_consensus.cc:374] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.978022 12112 ts_tablet_manager.cc:1397] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260927 23:04:44.978029 12113 raft_consensus.cc:729] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0061e7ce1e494432a56f67927fc03b66, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.978348 12113 consensus_queue.cc:260] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.978686 12112 raft_consensus.cc:348] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.978889 12112 raft_consensus.cc:374] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:44.979046 12112 raft_consensus.cc:729] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8cf5ee51c99463f9be1e92019449c31, State: Initialized, Role: FOLLOWER
I20260927 23:04:44.979466 12113 ts_tablet_manager.cc:1428] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260927 23:04:44.979489 12112 consensus_queue.cc:260] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:44.980072 12112 ts_tablet_manager.cc:1428] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260927 23:04:45.038584 12118 raft_consensus.cc:482] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:04:45.038769 12118 raft_consensus.cc:504] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:45.039407 12118 leader_election.cc:288] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051), 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961)
I20260927 23:04:45.039757 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ceb94a107b4b08b152b385336f2f0c" candidate_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8cf5ee51c99463f9be1e92019449c31" is_pre_election: true
I20260927 23:04:45.039994 11759 raft_consensus.cc:2454] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8554dbb06e544027bdd8ede9b87f5cb2 in term 0.
I20260927 23:04:45.039955 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ceb94a107b4b08b152b385336f2f0c" candidate_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0061e7ce1e494432a56f67927fc03b66" is_pre_election: true
I20260927 23:04:45.040174 11903 raft_consensus.cc:2454] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8554dbb06e544027bdd8ede9b87f5cb2 in term 0.
I20260927 23:04:45.040372 11968 leader_election.cc:302] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8554dbb06e544027bdd8ede9b87f5cb2, c8cf5ee51c99463f9be1e92019449c31; no voters:
I20260927 23:04:45.040655 12118 raft_consensus.cc:2781] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:04:45.040741 12118 raft_consensus.cc:482] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:04:45.040824 12118 raft_consensus.cc:3037] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:45.042568 12118 raft_consensus.cc:504] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:45.043104 12118 leader_election.cc:288] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [CANDIDATE]: Term 1 election: Requested vote from peers c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051), 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961)
I20260927 23:04:45.043503 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ceb94a107b4b08b152b385336f2f0c" candidate_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8cf5ee51c99463f9be1e92019449c31"
I20260927 23:04:45.043578 11903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ceb94a107b4b08b152b385336f2f0c" candidate_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0061e7ce1e494432a56f67927fc03b66"
I20260927 23:04:45.043704 11759 raft_consensus.cc:3037] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:45.043766 11903 raft_consensus.cc:3037] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:45.046059 11903 raft_consensus.cc:2454] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8554dbb06e544027bdd8ede9b87f5cb2 in term 1.
I20260927 23:04:45.046463 11971 leader_election.cc:302] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0061e7ce1e494432a56f67927fc03b66, 8554dbb06e544027bdd8ede9b87f5cb2; no voters:
I20260927 23:04:45.046655 11759 raft_consensus.cc:2454] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8554dbb06e544027bdd8ede9b87f5cb2 in term 1.
I20260927 23:04:45.046697 12118 raft_consensus.cc:2781] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:45.046952 12118 raft_consensus.cc:686] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 8554dbb06e544027bdd8ede9b87f5cb2, State: Running, Role: LEADER
I20260927 23:04:45.047235 12118 consensus_queue.cc:237] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } }
I20260927 23:04:45.049984 11602 catalog_manager.cc:5495] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195). New cstate: current_term: 1 leader_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8554dbb06e544027bdd8ede9b87f5cb2" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33901 } health_report { overall_health: HEALTHY } } }
I20260927 23:04:45.091835 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:45.086309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.3.240.195:33901)
0927 23:04:45.086894 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:45.086902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:45.086922 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:04:45.086982 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:45.087009 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:45.087091 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:45.087233 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:45.087590 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.088678 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.089945 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.090553 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.090682 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:45.091096 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:45.091235 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:45.091329 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:45.091465 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 23:04:45.091541 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":357,"threads_started":1}
I20260927 23:04:45.105739 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:45.099405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:45657 (local address 127.3.240.195:33901)
0927 23:04:45.099610 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:45.099618 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:45.099961 (+ 343us) server_negotiation.cc:407] Connection header received
0927 23:04:45.100135 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:45.100138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:45.100257 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:45.100319 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:04:45.100771 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.101640 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.103548 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.104132 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.104249 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:45.104888 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:04:45.105016 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:04:45.105100 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:45.105278 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 23:04:45.105371 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:04:45.106101 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:45.099316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33901 (local address 127.3.240.195:45657)
0927 23:04:45.099855 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:45.099873 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:45.100023 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:45.100356 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:45.100366 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:45.100378 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0927 23:04:45.100640 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:45.100650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.101763 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.101769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:45.103394 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:45.103410 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.104351 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.104360 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:45.104603 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:45.104663 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0927 23:04:45.105126 (+ 463us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0927 23:04:45.105139 (+ 13us) client_negotiation.cc:769] Sending connection context
0927 23:04:45.105234 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 23:04:45.105385 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":292,"threads_started":1}
I20260927 23:04:45.115243 11759 raft_consensus.cc:1261] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Refusing update from remote peer 8554dbb06e544027bdd8ede9b87f5cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:45.115243 11903 raft_consensus.cc:1261] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Refusing update from remote peer 8554dbb06e544027bdd8ede9b87f5cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:04:45.115708 12203 consensus_queue.cc:1035] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8cf5ee51c99463f9be1e92019449c31" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:45.116027 12156 consensus_queue.cc:1035] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0061e7ce1e494432a56f67927fc03b66" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:04:45.166906 11602 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:45.167191 11602 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:04:45.170019 11602 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d25edc8ffc94b4ca88a23777036bf2b: Sending DeleteTablet for 3 replicas of tablet f4ceb94a107b4b08b152b385336f2f0c
I20260927 23:04:45.170682 11739 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ceb94a107b4b08b152b385336f2f0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:45 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:45062
I20260927 23:04:45.170717 11878 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ceb94a107b4b08b152b385336f2f0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:45 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41868
I20260927 23:04:45.171020 12173 tablet_replica.cc:330] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: stopping tablet replica
I20260927 23:04:45.171180 12173 raft_consensus.cc:2229] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:45.171358 12173 raft_consensus.cc:2258] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:45.171629 12175 tablet_replica.cc:330] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: stopping tablet replica
I20260927 23:04:45.171784 12175 raft_consensus.cc:2229] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:04:45.171967 12175 raft_consensus.cc:2258] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:45.172900 12173 ts_tablet_manager.cc:1905] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:45.178568 12017 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ceb94a107b4b08b152b385336f2f0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:04:45 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39596
I20260927 23:04:45.178840 12174 tablet_replica.cc:330] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: stopping tablet replica
I20260927 23:04:45.179044 12174 raft_consensus.cc:2229] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:04:45.179440 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 11673
I20260927 23:04:45.179517 12174 raft_consensus.cc:2258] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:04:45.182798 12174 ts_tablet_manager.cc:1905] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:45.186669 12175 ts_tablet_manager.cc:1905] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:04:45.193032 12175 ts_tablet_manager.cc:1918] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:45.193243 12175 log.cc:1196] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/f4ceb94a107b4b08b152b385336f2f0c
I20260927 23:04:45.193329 12173 ts_tablet_manager.cc:1918] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:45.193451 12173 log.cc:1196] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/f4ceb94a107b4b08b152b385336f2f0c
I20260927 23:04:45.193814 12173 ts_tablet_manager.cc:1939] T f4ceb94a107b4b08b152b385336f2f0c P c8cf5ee51c99463f9be1e92019449c31: Deleting consensus metadata
I20260927 23:04:45.193850 12175 ts_tablet_manager.cc:1939] T f4ceb94a107b4b08b152b385336f2f0c P 0061e7ce1e494432a56f67927fc03b66: Deleting consensus metadata
I20260927 23:04:45.194815 11586 catalog_manager.cc:4841] TS c8cf5ee51c99463f9be1e92019449c31 (127.3.240.193:36051): tablet f4ceb94a107b4b08b152b385336f2f0c (table test-table [id=9334ae04147645f6859927c4f4d8f02f]) successfully deleted
I20260927 23:04:45.195231 11589 catalog_manager.cc:4841] TS 0061e7ce1e494432a56f67927fc03b66 (127.3.240.194:33961): tablet f4ceb94a107b4b08b152b385336f2f0c (table test-table [id=9334ae04147645f6859927c4f4d8f02f]) successfully deleted
I20260927 23:04:45.196564 12174 ts_tablet_manager.cc:1918] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260927 23:04:45.196722 12174 log.cc:1196] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/f4ceb94a107b4b08b152b385336f2f0c
I20260927 23:04:45.197098 12174 ts_tablet_manager.cc:1939] T f4ceb94a107b4b08b152b385336f2f0c P 8554dbb06e544027bdd8ede9b87f5cb2: Deleting consensus metadata
I20260927 23:04:45.200254 11588 catalog_manager.cc:4841] TS 8554dbb06e544027bdd8ede9b87f5cb2 (127.3.240.195:33901): tablet f4ceb94a107b4b08b152b385336f2f0c (table test-table [id=9334ae04147645f6859927c4f4d8f02f]) successfully deleted
Sep 27 23:04:45 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.193@KRBTEST.COM
I20260927 23:04:45.217371 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:45.180625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45078 (local address 127.3.240.193:36051)
0927 23:04:45.202236 (+ 21611us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:45.202244 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:45.202262 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:04:45.202312 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:45.202317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:45.202388 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:45.202510 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:45.204391 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.205239 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.206870 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.207336 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.207439 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:45.212011 (+ 4572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:45.212025 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:45.212040 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:45.212075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:45.214796 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:45.215451 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:45.215458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:45.215460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:45.215574 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:45.215929 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:45.215932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:45.215934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:45.216287 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:45.216590 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:45.216851 (+ 261us) server_negotiation.cc:299] Negotiation successful
0927 23:04:45.217065 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21495,"thread_start_us":21397,"threads_started":1}
I20260927 23:04:45.217991 11804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:45.358486 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 11673
I20260927 23:04:45.381471 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 11813
Sep 27 23:04:45 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.194@KRBTEST.COM
I20260927 23:04:45.399636 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:45.382376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41896 (local address 127.3.240.194:33961)
0927 23:04:45.382549 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:45.382554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:45.382570 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:04:45.382756 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:45.382760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:45.382820 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:45.382916 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:45.384717 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.385460 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.387421 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.387825 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.387953 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:45.393379 (+ 5426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:45.393395 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:45.393415 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:45.393458 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:45.396424 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:45.397410 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:45.397418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:45.397420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:45.397548 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:45.398058 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:45.398061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:45.398063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:45.398435 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:45.398662 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:45.399176 (+ 514us) server_negotiation.cc:299] Negotiation successful
0927 23:04:45.399339 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:04:45.400512 11943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:45.521101 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 11813
I20260927 23:04:45.544451 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 11952
Sep 27 23:04:45 dist-test-slave-lmfp krb5kdc[11557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.195@KRBTEST.COM
I20260927 23:04:45.562781 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:45.545472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39624 (local address 127.3.240.195:33901)
0927 23:04:45.545644 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:45.545648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:45.545664 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:04:45.545845 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:45.545848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:45.545952 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:45.546081 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:45.547931 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.548631 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.550529 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:45.550970 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:45.551238 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:45.556761 (+ 5523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:45.556776 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:45.556792 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:45.556825 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:45.559936 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:45.560610 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:45.560618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:45.560620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:45.560753 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:45.561147 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:45.561154 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:45.561161 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:45.561575 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:45.561928 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:45.562181 (+ 253us) server_negotiation.cc:299] Negotiation successful
0927 23:04:45.562424 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:04:45.563549 12082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:45.711683 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 11952
I20260927 23:04:45.711726 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:45.709931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36051 (local address 127.3.240.195:48341)
0927 23:04:45.711368 (+ 1437us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:45.711523 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:36051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1305,"thread_start_us":1159,"threads_started":1}
I20260927 23:04:45.736948 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 11570
I20260927 23:04:45.738194 11642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:45.852067 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 11570
2026-09-27T23:04:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6319 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:45 dist-test-slave-lmfp krb5kdc[12217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:45 dist-test-slave-lmfp krb5kdc[12217](info): set up 2 sockets
Sep 27 23:04:45 dist-test-slave-lmfp krb5kdc[12217](info): commencing operation
krb5kdc: starting...
W20260927 23:04:46.915890 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.011s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:47 dist-test-slave-lmfp krb5kdc[12217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:04:47.110433 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38541
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:47.419790 12230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:47.420104 12230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:47.420277 12230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:47.427093 12230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:47.427204 12230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:47.427279 12230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:47.427371 12230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:47.427461 12230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:47.436535 12230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:47.437704 12230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:47.438930 12230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:47.450045 12235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:47.450130 12236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:47.450102 12238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:47.451238 12230 server_base.cc:1034] running on GCE node
I20260927 23:04:47.451894 12230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:47.453094 12230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:47.454300 12230 hybrid_clock.cc:648] HybridClock initialized: now 1790550287454269 us; error 42 us; skew 500 ppm
I20260927 23:04:47.454630 12230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-27T23:04:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1663 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:04:47 dist-test-slave-lmfp krb5kdc[12244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:04:47 dist-test-slave-lmfp krb5kdc[12244](info): set up 2 sockets
Sep 27 23:04:47 dist-test-slave-lmfp krb5kdc[12244](info): commencing operation
krb5kdc: starting...
W20260927 23:04:48.579900 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:04:48 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:04:48.782136 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38567
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:49.082787 12257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:49.083053 12257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:49.083209 12257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:49.090193 12257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:49.090319 12257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:49.090402 12257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:49.090508 12257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:49.090590 12257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:49.099807 12257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:49.100973 12257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:49.102171 12257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:49.114578 12262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:49.114497 12263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:49.114527 12265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:49.115617 12257 server_base.cc:1034] running on GCE node
I20260927 23:04:49.116285 12257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:49.117638 12257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:49.118858 12257 hybrid_clock.cc:648] HybridClock initialized: now 1790550289118808 us; error 33 us; skew 500 ppm
I20260927 23:04:49.119269 12257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:49 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550289, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:49.125969 12257 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:04:49.127673 12257 webserver.cc:460] Webserver started at http://127.3.240.254:37851/ using document root <none> and password file <none>
I20260927 23:04:49.128355 12257 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:49.128454 12257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:49.128724 12257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:49.131817 12257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "15418724b6e645aca32d7da0629051db"
format_stamp: "Formatted at 2026-09-27 23:04:49 on dist-test-slave-lmfp"
server_key: "3b1f82bbde937fdfd6763e39ebbe2e24"
server_key_iv: "70e201abddcf76898d634c8fa250e78f"
server_key_version: "encryptionkey@0"
I20260927 23:04:49.132550 12257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "15418724b6e645aca32d7da0629051db"
format_stamp: "Formatted at 2026-09-27 23:04:49 on dist-test-slave-lmfp"
server_key: "3b1f82bbde937fdfd6763e39ebbe2e24"
server_key_iv: "70e201abddcf76898d634c8fa250e78f"
server_key_version: "encryptionkey@0"
I20260927 23:04:49.137606 12257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:04:49.141176 12272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:49.142477 12257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:04:49.142637 12257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "15418724b6e645aca32d7da0629051db"
format_stamp: "Formatted at 2026-09-27 23:04:49 on dist-test-slave-lmfp"
server_key: "3b1f82bbde937fdfd6763e39ebbe2e24"
server_key_iv: "70e201abddcf76898d634c8fa250e78f"
server_key_version: "encryptionkey@0"
I20260927 23:04:49.142788 12257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:49.161369 12257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:49.165127 12257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:49.165331 12257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:49.183035 12257 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:38567
I20260927 23:04:49.183077 12323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:38567 every 8 connection(s)
I20260927 23:04:49.184394 12257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:49.187985 12324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:04:49.190567 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 12257
I20260927 23:04:49.190780 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:04:49.191468 4035 external_mini_cluster.cc:1428] Setting key 1135a891f4b955f5fc5c1413c194040e
I20260927 23:04:49.199962 12324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Bootstrap starting.
I20260927 23:04:49.203768 12324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Neither blocks nor log segments found. Creating new log.
I20260927 23:04:49.205127 12324 log.cc:824] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Log is configured to *not* fsync() on all Append() calls
Sep 27 23:04:49 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:49.208235 12324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: No bootstrap required, opened a new log
I20260927 23:04:49.214233 12327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:49.193316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46188 (local address 127.3.240.254:38567)
0927 23:04:49.194088 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:49.194117 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:49.194156 (+ 39us) server_negotiation.cc:407] Connection header received
0927 23:04:49.195812 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:49.195834 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:49.196225 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:49.196537 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:49.198382 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:49.199758 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:49.201417 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:49.202026 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:49.202144 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:49.207366 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:49.207373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:49.207385 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:49.207421 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:49.210369 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:49.211226 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:49.211231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:49.211237 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:49.211370 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:49.211762 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:49.211765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:49.211766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:49.212240 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:49.212429 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:49.212910 (+ 481us) server_negotiation.cc:299] Negotiation successful
0927 23:04:49.213245 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":365,"threads_started":1}
I20260927 23:04:49.217453 12324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } }
I20260927 23:04:49.217839 12324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:04:49.217947 12324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15418724b6e645aca32d7da0629051db, State: Initialized, Role: FOLLOWER
I20260927 23:04:49.218547 12324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } }
I20260927 23:04:49.218842 12324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:49.218986 12324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:49.219202 12324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:04:49.221328 12324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } }
I20260927 23:04:49.221813 12324 leader_election.cc:302] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15418724b6e645aca32d7da0629051db; no voters:
I20260927 23:04:49.222527 12324 leader_election.cc:288] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:04:49.222707 12329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:04:49.224023 12329 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 1 LEADER]: Becoming Leader. State: Replica: 15418724b6e645aca32d7da0629051db, State: Running, Role: LEADER
I20260927 23:04:49.224495 12329 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } }
I20260927 23:04:49.224864 12324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:49.227129 12331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15418724b6e645aca32d7da0629051db. Latest consensus state: current_term: 1 leader_uuid: "15418724b6e645aca32d7da0629051db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } } }
I20260927 23:04:49.227182 12330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15418724b6e645aca32d7da0629051db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } } }
I20260927 23:04:49.227628 12331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:49.227628 12330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:49.228612 12338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:49.235327 12338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:49.243855 12338 catalog_manager.cc:1347] Generated new cluster ID: ea51a9adab4a42fe81f6a5f3a48bbf69
I20260927 23:04:49.243944 12338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:49.255497 12338 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:04:49.256673 12338 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:49.266345 12338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Generated new TSK 0
I20260927 23:04:49.267145 12338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:04:49.320638 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38567
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:49.621320 12350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:49.621600 12350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:49.621773 12350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:49.628350 12350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:49.628477 12350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:49.628621 12350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:49.638186 12350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:49.639484 12350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:49.640713 12350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:49.652906 12356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:49.653029 12358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:49.653077 12355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:49.653563 12350 server_base.cc:1034] running on GCE node
I20260927 23:04:49.654155 12350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:49.655063 12350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:49.656262 12350 hybrid_clock.cc:648] HybridClock initialized: now 1790550289656219 us; error 37 us; skew 500 ppm
I20260927 23:04:49.656541 12350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:49 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550289, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:49.659488 12350 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:04:49.660840 12350 webserver.cc:460] Webserver started at http://127.3.240.193:45687/ using document root <none> and password file <none>
I20260927 23:04:49.661489 12350 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:49.661621 12350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:49.661931 12350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:49.664659 12350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "91cd63feb1fa412a9db14be0f2834978"
format_stamp: "Formatted at 2026-09-27 23:04:49 on dist-test-slave-lmfp"
server_key: "a008391b3d9918d9448c175561d9310f"
server_key_iv: "8f784decebd8587d46fb0dc45968da65"
server_key_version: "encryptionkey@0"
I20260927 23:04:49.665373 12350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "91cd63feb1fa412a9db14be0f2834978"
format_stamp: "Formatted at 2026-09-27 23:04:49 on dist-test-slave-lmfp"
server_key: "a008391b3d9918d9448c175561d9310f"
server_key_iv: "8f784decebd8587d46fb0dc45968da65"
server_key_version: "encryptionkey@0"
I20260927 23:04:49.670377 12350 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260927 23:04:49.673607 12365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:49.674790 12350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 23:04:49.674962 12350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "91cd63feb1fa412a9db14be0f2834978"
format_stamp: "Formatted at 2026-09-27 23:04:49 on dist-test-slave-lmfp"
server_key: "a008391b3d9918d9448c175561d9310f"
server_key_iv: "8f784decebd8587d46fb0dc45968da65"
server_key_version: "encryptionkey@0"
I20260927 23:04:49.675216 12350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:49.692451 12350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:49.696175 12350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:49.696432 12350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:49.697299 12350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:49.698503 12350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:49.698616 12350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:49.698752 12350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:49.698827 12350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:49.729977 12350 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:39691
I20260927 23:04:49.730017 12477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:39691 every 8 connection(s)
I20260927 23:04:49.731395 12350 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:49.740034 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 12350
I20260927 23:04:49.740219 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:04:49.740937 4035 external_mini_cluster.cc:1428] Setting key 8a22133117b332f36ea63d7f4bf31b25
Sep 27 23:04:49 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550289, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:49.751827 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:49.733714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:43199 (local address 127.3.240.254:38567)
0927 23:04:49.734111 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:49.734116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:49.735090 (+ 974us) server_negotiation.cc:407] Connection header received
0927 23:04:49.736748 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:49.736756 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:49.736841 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:49.736963 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:49.738325 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:49.739039 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:49.741086 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:49.741566 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:49.741725 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:49.745928 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:49.745932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:49.745936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:49.745964 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:49.748418 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:49.749334 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:49.749338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:49.749340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:49.749455 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:49.749944 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:49.749951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:49.749955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:49.750326 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:49.750475 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:49.751081 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 23:04:49.751312 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":215,"threads_started":1}
I20260927 23:04:49.752534 12481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:49.734262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:43199)
0927 23:04:49.734896 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:49.734937 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:49.736495 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:49.737179 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:49.737186 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:49.737670 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:49.738173 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:49.738188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:49.739193 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:49.739196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:49.740914 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:49.740929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:49.742038 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:49.742044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:49.742309 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:49.743122 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:49.743148 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:49.745719 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:49.748556 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:49.748567 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:49.748586 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:49.749212 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:49.749560 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:49.749568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:49.749575 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:49.749806 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:49.750445 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:49.750450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:49.750734 (+ 284us) client_negotiation.cc:769] Sending connection context
0927 23:04:49.750960 (+ 226us) client_negotiation.cc:240] Negotiation successful
0927 23:04:49.751369 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":246,"threads_started":1}
I20260927 23:04:49.754171 12478 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38567
I20260927 23:04:49.754465 12478 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:49.755208 12478 heartbeater.cc:511] Master 127.3.240.254:38567 requested a full tablet report, sending...
I20260927 23:04:49.757380 12289 ts_manager.cc:194] Registered new tserver with Master: 91cd63feb1fa412a9db14be0f2834978 (127.3.240.193:39691)
I20260927 23:04:49.759605 12289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:43199
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:04:49.793514 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38567
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:50.100939 12484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:50.101152 12484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:50.101297 12484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:50.107903 12484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:50.108007 12484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:50.108175 12484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:50.116302 12484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:50.117291 12484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:50.118407 12484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:50.130606 12490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:50.130766 12489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:50.130741 12492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:50.132270 12484 server_base.cc:1034] running on GCE node
I20260927 23:04:50.132774 12484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:50.133599 12484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:50.134789 12484 hybrid_clock.cc:648] HybridClock initialized: now 1790550290134744 us; error 42 us; skew 500 ppm
I20260927 23:04:50.135136 12484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:50 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550290, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:50.137975 12484 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:04:50.139380 12484 webserver.cc:460] Webserver started at http://127.3.240.194:46317/ using document root <none> and password file <none>
I20260927 23:04:50.140007 12484 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:50.140139 12484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:50.140432 12484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:50.143224 12484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "fcb164a027f1420bbd5c9ffc5e94281e"
format_stamp: "Formatted at 2026-09-27 23:04:50 on dist-test-slave-lmfp"
server_key: "64560ad4e76c0e0b20144aae5eaef331"
server_key_iv: "8d374571f3b2b0598a87554297b082a6"
server_key_version: "encryptionkey@0"
I20260927 23:04:50.143954 12484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "fcb164a027f1420bbd5c9ffc5e94281e"
format_stamp: "Formatted at 2026-09-27 23:04:50 on dist-test-slave-lmfp"
server_key: "64560ad4e76c0e0b20144aae5eaef331"
server_key_iv: "8d374571f3b2b0598a87554297b082a6"
server_key_version: "encryptionkey@0"
I20260927 23:04:50.148727 12484 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:04:50.151741 12499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:50.152889 12484 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:04:50.153049 12484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "fcb164a027f1420bbd5c9ffc5e94281e"
format_stamp: "Formatted at 2026-09-27 23:04:50 on dist-test-slave-lmfp"
server_key: "64560ad4e76c0e0b20144aae5eaef331"
server_key_iv: "8d374571f3b2b0598a87554297b082a6"
server_key_version: "encryptionkey@0"
I20260927 23:04:50.153232 12484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:50.178772 12484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:50.182312 12484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:50.182650 12484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:50.183575 12484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:50.184844 12484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:50.184938 12484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:50.185078 12484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:50.185145 12484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:50.216274 12484 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:33917
I20260927 23:04:50.216313 12611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:33917 every 8 connection(s)
I20260927 23:04:50.217466 12484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:50.221694 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 12484
I20260927 23:04:50.221837 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:04:50.222244 4035 external_mini_cluster.cc:1428] Setting key 4e7c20fecd4624210a3e60847484d91b
Sep 27 23:04:50 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550290, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:50.237084 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:50.219868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:35565 (local address 127.3.240.254:38567)
0927 23:04:50.220043 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:50.220048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:50.221127 (+ 1079us) server_negotiation.cc:407] Connection header received
0927 23:04:50.222895 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:50.222900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:50.222982 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:50.223122 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:50.224439 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.225155 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.226670 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.227091 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.227211 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:50.230519 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:50.230549 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:50.230557 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:50.230597 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:50.233973 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:50.234647 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:50.234655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:50.234661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:50.234807 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:50.235282 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:50.235289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:50.235295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:50.235656 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:50.235800 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:50.236400 (+ 600us) server_negotiation.cc:299] Negotiation successful
0927 23:04:50.236636 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:04:50.237744 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:50.220347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:35565)
0927 23:04:50.220978 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:50.221021 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:50.222662 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:50.223330 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:50.223338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:50.223775 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:50.224274 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:50.224287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.225292 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.225296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:50.226538 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:50.226552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.227288 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.227296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:50.227501 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:50.228116 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:50.228136 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:50.230312 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:50.234097 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:50.234103 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:50.234115 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:50.234533 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:50.234930 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:50.234933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:50.234935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:50.235181 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:50.235776 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:50.235781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:50.236069 (+ 288us) client_negotiation.cc:769] Sending connection context
0927 23:04:50.236281 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 23:04:50.236619 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":243,"threads_started":1}
I20260927 23:04:50.239209 12612 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38567
I20260927 23:04:50.239502 12612 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:50.240233 12612 heartbeater.cc:511] Master 127.3.240.254:38567 requested a full tablet report, sending...
I20260927 23:04:50.241557 12289 ts_manager.cc:194] Registered new tserver with Master: fcb164a027f1420bbd5c9ffc5e94281e (127.3.240.194:33917)
I20260927 23:04:50.242295 12289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:35565
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:04:50.267632 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38567
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:50.576573 12617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:50.576839 12617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:50.576930 12617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:50.584012 12617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:50.584108 12617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:50.584211 12617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:50.593580 12617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:50.595202 12617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:50.596483 12617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:50.609498 12623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:50.609493 12625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:50.609611 12622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:50.610124 12617 server_base.cc:1034] running on GCE node
I20260927 23:04:50.610673 12617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:50.611544 12617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:50.612795 12617 hybrid_clock.cc:648] HybridClock initialized: now 1790550290612757 us; error 43 us; skew 500 ppm
I20260927 23:04:50.613128 12617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:50 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550290, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:50.616176 12617 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:04:50.617620 12617 webserver.cc:460] Webserver started at http://127.3.240.195:41439/ using document root <none> and password file <none>
I20260927 23:04:50.618295 12617 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:50.618439 12617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:50.618779 12617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:04:50.621644 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "7861e1c0991140e9a763ef6a1232ac99"
format_stamp: "Formatted at 2026-09-27 23:04:50 on dist-test-slave-lmfp"
server_key: "a2f601f11c51a9ea7cfecdd06d7453b7"
server_key_iv: "519d76bd7e0157ea3d9240a8151bc85a"
server_key_version: "encryptionkey@0"
I20260927 23:04:50.622416 12617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "7861e1c0991140e9a763ef6a1232ac99"
format_stamp: "Formatted at 2026-09-27 23:04:50 on dist-test-slave-lmfp"
server_key: "a2f601f11c51a9ea7cfecdd06d7453b7"
server_key_iv: "519d76bd7e0157ea3d9240a8151bc85a"
server_key_version: "encryptionkey@0"
I20260927 23:04:50.627558 12617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 23:04:50.630833 12632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:50.632123 12617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:50.632288 12617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "7861e1c0991140e9a763ef6a1232ac99"
format_stamp: "Formatted at 2026-09-27 23:04:50 on dist-test-slave-lmfp"
server_key: "a2f601f11c51a9ea7cfecdd06d7453b7"
server_key_iv: "519d76bd7e0157ea3d9240a8151bc85a"
server_key_version: "encryptionkey@0"
I20260927 23:04:50.632485 12617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:50.651721 12617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:50.655539 12617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:50.655833 12617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:50.656824 12617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:50.658571 12617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:50.658700 12617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:50.658864 12617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:50.658941 12617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:50.693518 12617 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:39321
I20260927 23:04:50.693581 12744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:39321 every 8 connection(s)
I20260927 23:04:50.694797 12617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:50.704612 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 12617
I20260927 23:04:50.704811 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:04:50.705240 4035 external_mini_cluster.cc:1428] Setting key 88dc2bdb367b83c056d4e7fa475e799d
Sep 27 23:04:50 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550290, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:04:50.714396 12480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:50.697350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:51011 (local address 127.3.240.254:38567)
0927 23:04:50.697517 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:50.697522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:50.698629 (+ 1107us) server_negotiation.cc:407] Connection header received
0927 23:04:50.700368 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:50.700374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:50.700444 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:50.700550 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:50.702042 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.702903 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.704560 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.705188 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.705297 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:50.709356 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:50.709360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:50.709364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:50.709394 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:50.711847 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:50.712504 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:50.712509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:50.712511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:50.712632 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:50.712991 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:50.712994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:50.712996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:50.713230 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:50.713331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:50.714027 (+ 696us) server_negotiation.cc:299] Negotiation successful
0927 23:04:50.714169 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:04:50.715459 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:50.697783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:51011)
0927 23:04:50.698456 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:50.698502 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:50.700107 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:50.700740 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:50.700747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:50.701249 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:50.701795 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:50.701822 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.703091 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.703095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:50.704380 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:50.704397 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.705423 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.705431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:50.705711 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:50.706456 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:50.706493 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:50.709145 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:04:50.711967 (+ 2822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:50.711974 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:50.711984 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:50.712360 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:50.712722 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:04:50.712725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:04:50.712727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:04:50.712875 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:04:50.713365 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:04:50.713371 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:04:50.713695 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 23:04:50.713952 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 23:04:50.714308 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":254,"threads_started":1}
I20260927 23:04:50.716713 12745 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38567
I20260927 23:04:50.716996 12745 heartbeater.cc:464] Registering TS with master...
I20260927 23:04:50.717685 12745 heartbeater.cc:511] Master 127.3.240.254:38567 requested a full tablet report, sending...
I20260927 23:04:50.719337 12289 ts_manager.cc:194] Registered new tserver with Master: 7861e1c0991140e9a763ef6a1232ac99 (127.3.240.195:39321)
I20260927 23:04:50.720444 12289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:51011
I20260927 23:04:50.720860 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.193@KRBTEST.COM" created.
I20260927 23:04:50.762442 12478 heartbeater.cc:503] Master 127.3.240.254:38567 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
WARNING: no policy specified for oryx/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
WARNING: no policy specified for oryx/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
I20260927 23:04:50.849329 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12350
Sep 27 23:04:50 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:04:50.866655 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:50.850541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.3.240.193:39691)
0927 23:04:50.851256 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:50.851266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:50.851559 (+ 293us) server_negotiation.cc:407] Connection header received
0927 23:04:50.851772 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:50.851779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:50.852010 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:50.852182 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:50.853530 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.854530 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.855817 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.856443 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.856581 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:50.860989 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:50.861001 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:50.861015 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:50.861047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:50.863934 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:50.864609 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:50.864616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:50.864619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:50.864738 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:50.865119 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:50.865124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:50.865126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:50.865446 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:50.865770 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:50.866054 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 23:04:50.866319 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":300,"threads_started":1}
I20260927 23:04:50.868002 12473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:50.950910 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 12350
I20260927 23:04:50.967803 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12484
Sep 27 23:04:50 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:04:50.983464 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:50.968669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.3.240.194:33917)
0927 23:04:50.969212 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:50.969218 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:50.969242 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:04:50.969301 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:50.969307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:50.969510 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:50.969673 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:50.970899 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.971984 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.973233 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:50.973778 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:50.973935 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:50.977914 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:50.977923 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:50.977936 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:50.977967 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:50.980614 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:50.981231 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:50.981247 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:50.981250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:50.981359 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:50.981714 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:50.981720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:50.981723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:50.982047 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:50.982355 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:50.982661 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 23:04:50.982921 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":283,"threads_started":1}
I20260927 23:04:50.984746 12607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:51.068264 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 12484
I20260927 23:04:51.083786 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12617
Sep 27 23:04:51 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:04:51.099428 12759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:51.084793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.3.240.195:39321)
0927 23:04:51.085428 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:51.085436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:51.085464 (+ 28us) server_negotiation.cc:407] Connection header received
0927 23:04:51.085522 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:51.085528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:51.085738 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:51.085869 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:51.087115 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:51.088097 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:51.089255 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:51.089787 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:51.089880 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:51.094024 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:51.094034 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:51.094048 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:51.094083 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:51.096883 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:51.097454 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:51.097461 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:51.097463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:51.097584 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:51.097975 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:51.097979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:51.097981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:51.098314 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:51.098557 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:51.098796 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 23:04:51.099094 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":334,"threads_started":1}
I20260927 23:04:51.100672 12740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:51.188267 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 12617
I20260927 23:04:51.206940 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12257
I20260927 23:04:51.208168 12319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:04:51.308687 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 12257
I20260927 23:04:51.329980 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38567
--webserver_interface=127.3.240.254
--webserver_port=37851
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:04:51.636207 12762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:51.636521 12762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:51.636677 12762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:51.643430 12762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:04:51.643567 12762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:51.643667 12762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:51.643764 12762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:04:51.643849 12762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:04:51.652933 12762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=37851
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:51.654189 12762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:51.655406 12762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:51.666591 12770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:51.666689 12768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:51.667014 12767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:51.668254 12762 server_base.cc:1034] running on GCE node
I20260927 23:04:51.668931 12762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:51.670162 12762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:51.671378 12762 hybrid_clock.cc:648] HybridClock initialized: now 1790550291671341 us; error 47 us; skew 500 ppm
I20260927 23:04:51.671701 12762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:51 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550291, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:51.674598 12762 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:04:51.676024 12762 webserver.cc:460] Webserver started at http://127.3.240.254:37851/ using document root <none> and password file <none>
I20260927 23:04:51.676658 12762 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:51.676797 12762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:51.682202 12762 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260927 23:04:51.684943 12777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:51.686105 12762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:04:51.686272 12762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "15418724b6e645aca32d7da0629051db"
format_stamp: "Formatted at 2026-09-27 23:04:49 on dist-test-slave-lmfp"
server_key: "3b1f82bbde937fdfd6763e39ebbe2e24"
server_key_iv: "70e201abddcf76898d634c8fa250e78f"
server_key_version: "encryptionkey@0"
I20260927 23:04:51.687240 12762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:51.715274 12762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:51.718886 12762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:51.719147 12762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:51.737661 12762 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:38567
I20260927 23:04:51.737697 12828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:38567 every 8 connection(s)
I20260927 23:04:51.738982 12762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:04:51.745764 12829 sys_catalog.cc:263] Verifying existing consensus state
I20260927 23:04:51.748658 12829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Bootstrap starting.
I20260927 23:04:51.749055 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 12762
I20260927 23:04:51.749750 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:39691
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:38567
--webserver_port=45687
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.240.193
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:04:51.766629 12829 log.cc:824] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Log is configured to *not* fsync() on all Append() calls
I20260927 23:04:51.775269 12829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:04:51.775776 12829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Bootstrap complete.
I20260927 23:04:51.784408 12829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } }
I20260927 23:04:51.785164 12829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15418724b6e645aca32d7da0629051db, State: Initialized, Role: FOLLOWER
I20260927 23:04:51.785763 12829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } }
I20260927 23:04:51.785997 12829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:04:51.786098 12829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:04:51.786279 12829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:04:51.789721 12829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } }
I20260927 23:04:51.790360 12829 leader_election.cc:302] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15418724b6e645aca32d7da0629051db; no voters:
I20260927 23:04:51.791100 12829 leader_election.cc:288] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 23:04:51.791338 12833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:04:51.795121 12833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [term 2 LEADER]: Becoming Leader. State: Replica: 15418724b6e645aca32d7da0629051db, State: Running, Role: LEADER
I20260927 23:04:51.795686 12833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } }
I20260927 23:04:51.795913 12829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:04:51.798550 12835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15418724b6e645aca32d7da0629051db. Latest consensus state: current_term: 2 leader_uuid: "15418724b6e645aca32d7da0629051db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } } }
I20260927 23:04:51.799551 12835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:51.799279 12834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "15418724b6e645aca32d7da0629051db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15418724b6e645aca32d7da0629051db" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38567 } } }
I20260927 23:04:51.799736 12834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db [sys.catalog]: This master's current role is: LEADER
I20260927 23:04:51.810283 12841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:04:51.816033 12841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:04:51.817451 12841 catalog_manager.cc:1259] Loaded cluster ID: ea51a9adab4a42fe81f6a5f3a48bbf69
I20260927 23:04:51.817582 12841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:04:51.821532 12841 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:04:51.823064 12841 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 15418724b6e645aca32d7da0629051db: Loaded TSK: 0
I20260927 23:04:51.824590 12841 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 23:04:52.079344 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:52.079646 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:52.079809 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:52.086761 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:52.086849 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:52.087061 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:04:52.096513 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:39691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=45687
--tserver_master_addrs=127.3.240.254:38567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:52.097831 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:52.099100 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:52.111456 12856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:52.111711 12858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:52.111450 12855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:52.112336 12831 server_base.cc:1034] running on GCE node
I20260927 23:04:52.113072 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:52.114231 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:52.115518 12831 hybrid_clock.cc:648] HybridClock initialized: now 1790550292115427 us; error 75 us; skew 500 ppm
I20260927 23:04:52.115881 12831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:52 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550292, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:52.119760 12831 init.cc:377] Logged in from keytab as oryx/127.3.240.193@KRBTEST.COM (short username oryx)
I20260927 23:04:52.121344 12831 webserver.cc:460] Webserver started at http://127.3.240.193:45687/ using document root <none> and password file <none>
I20260927 23:04:52.122005 12831 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:52.122134 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:52.127439 12831 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:52.130064 12865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:52.131433 12831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:52.131597 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "91cd63feb1fa412a9db14be0f2834978"
format_stamp: "Formatted at 2026-09-27 23:04:49 on dist-test-slave-lmfp"
server_key: "a008391b3d9918d9448c175561d9310f"
server_key_iv: "8f784decebd8587d46fb0dc45968da65"
server_key_version: "encryptionkey@0"
I20260927 23:04:52.132516 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:52.166411 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:52.170619 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:52.170876 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:52.171834 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:52.172984 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:52.173095 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:52.173211 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:52.173290 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:52.205592 12831 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:39691
I20260927 23:04:52.205682 12977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:39691 every 8 connection(s)
I20260927 23:04:52.206938 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:04:52.211465 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 12831
I20260927 23:04:52.212155 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:33917
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:38567
--webserver_port=46317
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.240.194
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:04:52 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:52 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 23:04:52.522054 12982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:52.522328 12982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:52.522464 12982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:52.529805 12982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:52.529903 12982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:52.530084 12982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:04:52.539192 12982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:33917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=46317
--tserver_master_addrs=127.3.240.254:38567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:52.540423 12982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:52.541635 12982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:52.558513 12989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:52.558928 12991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:52.559144 12982 server_base.cc:1034] running on GCE node
W20260927 23:04:52.559309 12988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:52.559986 12982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:52.560935 12982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:52.562177 12982 hybrid_clock.cc:648] HybridClock initialized: now 1790550292562095 us; error 50 us; skew 500 ppm
I20260927 23:04:52.562487 12982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:52 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550292, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:52.565822 12982 init.cc:377] Logged in from keytab as oryx/127.3.240.194@KRBTEST.COM (short username oryx)
I20260927 23:04:52.567373 12982 webserver.cc:460] Webserver started at http://127.3.240.194:46317/ using document root <none> and password file <none>
I20260927 23:04:52.567987 12982 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:52.568125 12982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:52.573653 12982 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:04:52.576779 12998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:52.578382 12982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:04:52.578552 12982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "fcb164a027f1420bbd5c9ffc5e94281e"
format_stamp: "Formatted at 2026-09-27 23:04:50 on dist-test-slave-lmfp"
server_key: "64560ad4e76c0e0b20144aae5eaef331"
server_key_iv: "8d374571f3b2b0598a87554297b082a6"
server_key_version: "encryptionkey@0"
I20260927 23:04:52.579519 12982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:04:52.598748 12982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:04:52.602806 12982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:52.603149 12982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:52.604146 12982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:52.605455 12982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:52.605568 12982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:52.605692 12982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:52.605772 12982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:52.648664 12982 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:33917
I20260927 23:04:52.648702 13110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:33917 every 8 connection(s)
I20260927 23:04:52.650013 12982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:04:52.652501 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 12982
I20260927 23:04:52.653443 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:39321
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:38567
--webserver_port=41439
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.240.195
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:04:52 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:52 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 23:04:53.322268 13115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:04:53.322674 13115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:04:53.322944 13115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:04:53.334852 13115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:04:53.335016 13115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:04:53.335345 13115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:04:53.352174 13115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:39321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=41439
--tserver_master_addrs=127.3.240.254:38567
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:04:53.353950 13115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:04:53.355675 13115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:04:53.375186 13121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:04:53.376051 13122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:53.384390 13115 server_base.cc:1034] running on GCE node
W20260927 23:04:53.384768 13124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:04:53.385839 13115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:04:53.387126 13115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:04:53.388464 13115 hybrid_clock.cc:648] HybridClock initialized: now 1790550293388336 us; error 84 us; skew 500 ppm
I20260927 23:04:53.388895 13115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550293, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:04:53.394207 13115 init.cc:377] Logged in from keytab as oryx/127.3.240.195@KRBTEST.COM (short username oryx)
I20260927 23:04:53.396955 13115 webserver.cc:460] Webserver started at http://127.3.240.195:41439/ using document root <none> and password file <none>
I20260927 23:04:53.398382 13115 fs_manager.cc:359] Metadata directory not provided
I20260927 23:04:53.398576 13115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:04:53.408631 13115 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260927 23:04:53.413383 13131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:53.415669 13115 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260927 23:04:53.415921 13115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "7861e1c0991140e9a763ef6a1232ac99"
format_stamp: "Formatted at 2026-09-27 23:04:50 on dist-test-slave-lmfp"
server_key: "a2f601f11c51a9ea7cfecdd06d7453b7"
server_key_iv: "519d76bd7e0157ea3d9240a8151bc85a"
server_key_version: "encryptionkey@0"
I20260927 23:04:53.417884 13115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260927 23:04:53.420399 12981 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.200s user 0.001s sys 0.011s
I20260927 23:04:53.422237 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:52.210006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:40477)
0927 23:04:52.210860 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:52.210898 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:52.213003 (+ 2105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:52.214338 (+ 1335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:52.214350 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:52.214951 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:52.215533 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:52.215549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:52.217103 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:52.217115 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:52.218289 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:52.218300 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:52.219375 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:52.219409 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:52.219655 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:52.220315 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:52.220339 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:53.421174 (+1200835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":327,"threads_started":1}
W20260927 23:04:53.422355 12980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:52.209920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:40477 (local address 127.3.240.254:38567)
0927 23:04:52.210689 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:52.210718 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:52.211171 (+ 453us) server_negotiation.cc:407] Connection header received
0927 23:04:52.213517 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:52.213528 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:52.213832 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:52.214200 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:52.215717 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:52.216940 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:52.218463 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:52.219156 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:52.219337 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.421596 (+1202259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:40477: BlockingRecv error: recv got EOF from 127.3.240.193:40477 (error 108)
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":266,"threads_started":1}
W20260927 23:04:53.423120 12978 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:38567 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.240.254:38567: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:53.437742 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:53.423807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:59781)
0927 23:04:53.424062 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:53.424081 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:53.424196 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:53.424795 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:53.424800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:53.425220 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:53.425430 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.425441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.427638 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.427643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.429093 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.429105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.430504 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.430510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.430673 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.431444 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:53.431487 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:53.437475 (+ 5988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144}
W20260927 23:04:53.438298 12980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:53.424079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:59781 (local address 127.3.240.254:38567)
0927 23:04:53.424329 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:53.424340 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:53.424370 (+ 30us) server_negotiation.cc:407] Connection header received
0927 23:04:53.424464 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:53.424473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:53.424587 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:53.424760 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:53.425597 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.427451 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.429257 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.430292 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.430482 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.437889 (+ 7407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:59781: BlockingRecv error: recv got EOF from 127.3.240.193:59781 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:53.456243 13115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260927 23:04:53.457525 12980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:53.439025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:40757 (local address 127.3.240.254:38567)
0927 23:04:53.439266 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:53.439277 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:53.443439 (+ 4162us) server_negotiation.cc:407] Connection header received
0927 23:04:53.443495 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:53.443499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:53.443575 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:53.443653 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:53.444953 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.446007 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.448964 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.449556 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.449679 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.457284 (+ 7605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:40757: BlockingRecv error: recv got EOF from 127.3.240.193:40757 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 23:04:53.457590 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:53.438524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:40757)
0927 23:04:53.443217 (+ 4693us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:53.443233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:53.443336 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:53.443771 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:53.443781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:53.444412 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:53.444737 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.444757 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.446192 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.446202 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.448751 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.448776 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.449754 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.449765 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.450147 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.451029 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:53.451072 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:53.457065 (+ 5993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4441,"thread_start_us":5191,"threads_started":1}
W20260927 23:04:53.458200 12978 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 23:04:53.461443 13115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:04:53.461721 13115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:04:53.462796 13115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:04:53.464239 13115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:04:53.464372 13115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:53.464491 13115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:04:53.464576 13115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:04:53.512948 13115 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:39321
I20260927 23:04:53.513032 13244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:39321 every 8 connection(s)
I20260927 23:04:53.514417 13115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:04:53.518543 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 13115
I20260927 23:04:53.534886 13249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:53.519936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.3.240.254:38567)
0927 23:04:53.520491 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:53.520495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:53.520506 (+ 11us) server_negotiation.cc:407] Connection header received
0927 23:04:53.520550 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:53.520553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:53.520604 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:53.520718 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:53.522121 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.522926 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.524511 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.524883 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.525009 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.528323 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:04:53.528330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:04:53.528344 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:04:53.528378 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:04:53.531290 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:53.532443 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:53.532449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:53.532455 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:53.532592 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:04:53.532934 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:04:53.532939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:04:53.532942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:04:53.533331 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:04:53.533670 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0927 23:04:53.534269 (+ 599us) server_negotiation.cc:299] Negotiation successful
0927 23:04:53.534590 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":296,"threads_started":1}
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
W20260927 23:04:53.777895 13114 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.103s user 0.005s sys 0.007s
I20260927 23:04:53.779484 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:52.653276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:52565)
0927 23:04:52.667787 (+ 14511us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:52.667831 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:52.669527 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:52.670256 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:52.670264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:52.670759 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:52.671357 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:52.671373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:52.672350 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:52.672357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:52.673444 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:52.673454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:52.674197 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:52.674207 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:52.674445 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:52.675106 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:52.675126 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:53.778586 (+1103460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14205,"thread_start_us":13981,"threads_started":1}
W20260927 23:04:53.779498 13113 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:52.652507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52565 (local address 127.3.240.254:38567)
0927 23:04:52.667365 (+ 14858us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:52.667373 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:52.667955 (+ 582us) server_negotiation.cc:407] Connection header received
0927 23:04:52.669791 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:52.669796 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:52.669878 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:52.670003 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:52.671522 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:52.672213 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:52.673596 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:52.673991 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:52.674118 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.778973 (+1104855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:52565: BlockingRecv error: recv got EOF from 127.3.240.194:52565 (error 108)
Metrics: {"server-negotiator.queue_time_us":14721,"thread_start_us":14613,"threads_started":1}
W20260927 23:04:53.780478 13111 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:38567 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.240.254:38567: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:53.792179 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:53.781227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:39233)
0927 23:04:53.781494 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:53.781508 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:53.781598 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:53.781963 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:53.781966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:53.782288 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:53.782472 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.782484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.784539 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.784542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.785445 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.785455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.786716 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.786723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.786881 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.787787 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:53.787809 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:53.791999 (+ 4190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173}
W20260927 23:04:53.792665 13113 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:53.781227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:39233 (local address 127.3.240.254:38567)
0927 23:04:53.781383 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:53.781388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:53.781594 (+ 206us) server_negotiation.cc:407] Connection header received
0927 23:04:53.781710 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:53.781718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:53.781823 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:53.781953 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:53.782637 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.784379 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.785580 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.786486 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.786659 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.792313 (+ 5654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:39233: BlockingRecv error: recv got EOF from 127.3.240.194:39233 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:53 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:53.804296 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:53.792844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:47415)
0927 23:04:53.793119 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:53.793136 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:53.793222 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:53.793616 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:53.793619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:53.793863 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:53.793992 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.793999 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.795885 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.795889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.796891 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.796905 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.798255 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.798259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.798380 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.799088 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:53.799110 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:53.804116 (+ 5006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178}
W20260927 23:04:53.804661 13111 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 23:04:53.804910 13113 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:53.793008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:47415 (local address 127.3.240.254:38567)
0927 23:04:53.793247 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:53.793257 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:53.793281 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:04:53.793346 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:53.793354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:53.793471 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:53.793606 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:53.794126 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.795723 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.797039 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.798088 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.798262 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.804494 (+ 6232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:47415: BlockingRecv error: recv got EOF from 127.3.240.194:47415 (error 108)
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":94}
Sep 27 23:04:54 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:54 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:54.479436 13252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:54.458921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:51245)
0927 23:04:54.461363 (+ 2442us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:54.461386 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:54.461521 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:54.462191 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:54.462194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:54.462583 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:54.462753 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:54.462761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.464772 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.464776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:54.465671 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:54.465681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.466942 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.466947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:54.467121 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.475206 (+ 8085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:54.475223 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:54.479196 (+ 3973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2284,"thread_start_us":2086,"threads_started":1}
W20260927 23:04:54.480177 13251 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:54.458944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:51245 (local address 127.3.240.254:38567)
0927 23:04:54.461785 (+ 2841us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:54.461794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:54.461822 (+ 28us) server_negotiation.cc:407] Connection header received
0927 23:04:54.461895 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:54.461903 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:54.462021 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:54.462179 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:54.462906 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.464604 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.465809 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.466746 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.466912 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.479795 (+ 12883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:51245: BlockingRecv error: recv got EOF from 127.3.240.193:51245 (error 108)
Metrics: {"server-negotiator.queue_time_us":2683,"thread_start_us":2514,"threads_started":1}
W20260927 23:04:54.606437 13247 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.072s user 0.010s sys 0.015s
W20260927 23:04:54.607563 12980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:53.517513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:44449 (local address 127.3.240.254:38567)
0927 23:04:53.517752 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:53.517758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:53.519243 (+ 1485us) server_negotiation.cc:407] Connection header received
0927 23:04:53.524158 (+ 4915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:53.524164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:53.524237 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:53.524332 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:53.527237 (+ 2905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.528129 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.531757 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.532239 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.532324 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.607363 (+1075039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:44449: BlockingRecv error: recv got EOF from 127.3.240.195:44449 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 23:04:54.608191 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:53.517825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:44449)
0927 23:04:53.518978 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:53.519087 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:53.523666 (+ 4579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:53.524690 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:53.524707 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:53.525782 (+ 1075us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:53.526972 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.527045 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.528311 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.528321 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.531565 (+ 3244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:53.531588 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:53.532508 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:53.532520 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:53.532985 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:53.533890 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:53.533939 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:54.607237 (+1073298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":442,"threads_started":1}
W20260927 23:04:54.609097 13245 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:38567 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.240.254:38567: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Sep 27 23:04:54 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:54 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:54.617656 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:54.609738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:47541)
0927 23:04:54.609918 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:54.609943 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:54.610030 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:54.610296 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:54.610299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:54.610638 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:54.610871 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:54.610882 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.612217 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.612224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:54.613276 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:54.613286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.613996 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.614000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:54.614115 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.614593 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:54.614608 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:54.617480 (+ 2872us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260927 23:04:54.617803 12980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:54.609794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47541 (local address 127.3.240.254:38567)
0927 23:04:54.609982 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:54.609986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:54.610006 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:04:54.610126 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:54.610129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:54.610186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:54.610264 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:54.611011 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.612111 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.613388 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.613839 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.613930 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.617632 (+ 3702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:47541: BlockingRecv error: recv got EOF from 127.3.240.195:47541 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 27 23:04:54 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:54 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:54.625476 13247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:54.618257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:55621)
0927 23:04:54.618384 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:54.618399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:54.618475 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:54.618701 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:54.618704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:54.618941 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:54.619153 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:54.619160 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.620081 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.620084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:54.620974 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:54.620983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.621537 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.621541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:54.621685 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.622120 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:54.622134 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:54.625313 (+ 3179us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 23:04:54.625710 12980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:54.618331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:55621 (local address 127.3.240.254:38567)
0927 23:04:54.618455 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:54.618459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:54.618472 (+ 13us) server_negotiation.cc:407] Connection header received
0927 23:04:54.618538 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:54.618541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:54.618595 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:54.618663 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:54.619265 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.619972 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.621073 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.621430 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.621518 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.625466 (+ 3948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:55621: BlockingRecv error: recv got EOF from 127.3.240.195:55621 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260927 23:04:54.625728 13245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 27 23:04:54 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:54 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:54.813907 13253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:54.805183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:33757)
0927 23:04:54.805708 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:54.805726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:54.805864 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:54.806303 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:54.806308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:54.806765 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:54.806913 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:54.806921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.807972 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.807976 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:54.809022 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:54.809037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.809732 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.809736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:54.809857 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.810498 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:54.810516 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:54.813666 (+ 3150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":276,"threads_started":1}
W20260927 23:04:54.814069 12980 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:54.805295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:33757 (local address 127.3.240.254:38567)
0927 23:04:54.805501 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:54.805509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:54.805794 (+ 285us) server_negotiation.cc:407] Connection header received
0927 23:04:54.806025 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:54.806030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:54.806129 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:54.806228 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:54.807086 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.807796 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.809150 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:54.809558 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:54.809648 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:54.813854 (+ 4206us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:33757: BlockingRecv error: recv got EOF from 127.3.240.194:33757 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 23:04:55 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:55 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:55.488976 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:55.480310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:57755)
0927 23:04:55.480795 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:55.480811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:55.480923 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:55.481173 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:55.481176 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:55.481545 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:55.481712 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:55.481720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.482783 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.482787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:55.483760 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:55.483823 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.484456 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.484459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:55.484574 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:55.485096 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:55.485111 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:55.488743 (+ 3632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":278,"threads_started":1}
W20260927 23:04:55.489105 13255 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:55.480342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57755 (local address 127.3.240.254:38567)
0927 23:04:55.480827 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:55.480844 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:55.480876 (+ 32us) server_negotiation.cc:407] Connection header received
0927 23:04:55.481006 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:55.481009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:55.481064 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:55.481139 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:55.481822 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.482590 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.483956 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.484343 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.484430 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:55.488899 (+ 4469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:57755: BlockingRecv error: recv got EOF from 127.3.240.193:57755 (error 108)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":278,"threads_started":1}
Sep 27 23:04:55 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:55 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:55.635066 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:55.626303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:44117)
0927 23:04:55.626847 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:55.626881 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:55.627036 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:55.627410 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:55.627416 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:55.627909 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:55.628048 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:55.628056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.629045 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.629048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:55.629964 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:55.629974 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.630677 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.630682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:55.630802 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:55.631417 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:55.631433 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:55.634769 (+ 3336us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":322,"threads_started":1}
W20260927 23:04:55.635221 13255 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:55.626394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:44117 (local address 127.3.240.254:38567)
0927 23:04:55.626719 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:55.626725 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:55.626969 (+ 244us) server_negotiation.cc:407] Connection header received
0927 23:04:55.627175 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:55.627180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:55.627262 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:55.627358 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:55.628187 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.628931 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.630106 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.630515 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.630615 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:55.634955 (+ 4340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:44117: BlockingRecv error: recv got EOF from 127.3.240.195:44117 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 27 23:04:55 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:55 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:55.823110 13257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:55.814757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:48317)
0927 23:04:55.815235 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:55.815249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:55.815387 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:55.815729 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:55.815733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:55.816134 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:55.816305 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:55.816314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.817255 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.817262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:55.818150 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:55.818160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.818848 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.818853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:55.819073 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:55.819641 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:55.819656 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:55.822720 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":289,"threads_started":1}
W20260927 23:04:55.823172 13255 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:55.814831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:48317 (local address 127.3.240.254:38567)
0927 23:04:55.815030 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:55.815037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:55.815321 (+ 284us) server_negotiation.cc:407] Connection header received
0927 23:04:55.815503 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:55.815507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:55.815571 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:55.815681 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:55.816446 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.817131 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.818284 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:55.818715 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:55.818838 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:55.822919 (+ 4081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:48317: BlockingRecv error: recv got EOF from 127.3.240.194:48317 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 23:04:56 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:56 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:56.498389 13258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:56.489986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:35065)
0927 23:04:56.490594 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:56.490610 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:56.490719 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:56.491090 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:56.491093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:56.491422 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:56.491609 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:56.491618 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.492566 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.492569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:56.493401 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:56.493412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.494021 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.494025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:56.494136 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:56.494655 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:56.494671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:56.498069 (+ 3398us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":281,"threads_started":1}
W20260927 23:04:56.498546 13259 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:56.490110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:35065 (local address 127.3.240.254:38567)
0927 23:04:56.490621 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:56.490625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:56.490659 (+ 34us) server_negotiation.cc:407] Connection header received
0927 23:04:56.490818 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:56.490821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:56.490874 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:56.490964 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:56.491745 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.492449 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.493515 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.493876 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.493995 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:56.498282 (+ 4287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:35065: BlockingRecv error: recv got EOF from 127.3.240.193:35065 (error 108)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":247,"threads_started":1}
Sep 27 23:04:56 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:56 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:56.644174 13260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:56.635881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:53949)
0927 23:04:56.636479 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:56.636505 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:56.636610 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:56.636964 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:56.636968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:56.637425 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:56.637599 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:56.637607 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.638557 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.638563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:56.639443 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:56.639454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.640122 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.640128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:56.640298 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:56.640826 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:56.640842 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:56.643968 (+ 3126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":380,"threads_started":1}
W20260927 23:04:56.644296 13259 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:56.635957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:53949 (local address 127.3.240.254:38567)
0927 23:04:56.636154 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:56.636160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:56.636570 (+ 410us) server_negotiation.cc:407] Connection header received
0927 23:04:56.636762 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:56.636766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:56.636822 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:56.636938 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:56.637738 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.638414 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.639576 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.639969 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.640088 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:56.644117 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:53949: BlockingRecv error: recv got EOF from 127.3.240.195:53949 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 27 23:04:56 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:56 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:56.832181 13261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:56.824061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:56089)
0927 23:04:56.824524 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:56.824538 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:56.824671 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:56.824993 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:56.824998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:56.825379 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:56.825551 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:56.825559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.826593 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.826600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:56.827569 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:56.827579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.828234 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.828239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:56.828397 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:56.828957 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:56.828973 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:56.831958 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":236,"threads_started":1}
W20260927 23:04:56.832397 13259 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:56.824105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:56089 (local address 127.3.240.254:38567)
0927 23:04:56.824289 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:56.824295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:56.824612 (+ 317us) server_negotiation.cc:407] Connection header received
0927 23:04:56.824792 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:56.824797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:56.824870 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:56.824963 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:56.825715 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.826472 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.827713 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:56.828107 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:56.828228 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:56.832135 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:56089: BlockingRecv error: recv got EOF from 127.3.240.194:56089 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 27 23:04:57 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:57 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:57.508091 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:57.499417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:55301)
0927 23:04:57.499904 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:57.499920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:57.500018 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:57.500441 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:57.500444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:57.500776 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:57.500934 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:57.500943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.501814 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.501818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:57.502803 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:57.502814 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.503449 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.503453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:57.503573 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:57.504197 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:57.504214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:57.507838 (+ 3624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":250,"threads_started":1}
W20260927 23:04:57.508378 13263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:57.499488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:55301 (local address 127.3.240.254:38567)
0927 23:04:57.500189 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:57.500193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:57.500207 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:04:57.500246 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:57.500249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:57.500304 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:57.500395 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:57.501051 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.501714 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.502935 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.503323 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.503421 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:57.508020 (+ 4599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:55301: BlockingRecv error: recv got EOF from 127.3.240.193:55301 (error 108)
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":506,"threads_started":1}
Sep 27 23:04:57 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:57 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:57.654627 13264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:57.645010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:40691)
0927 23:04:57.645586 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:57.645599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:57.645744 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:57.646062 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:57.646067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:57.646458 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:57.646619 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:57.646627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.647724 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.647729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:57.649123 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:57.649136 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.649768 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.649774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:57.649957 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:57.650577 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:57.650617 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:57.654399 (+ 3782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":257,"threads_started":1}
W20260927 23:04:57.654780 13263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:57.645214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:40691 (local address 127.3.240.254:38567)
0927 23:04:57.645498 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:57.645504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:57.645678 (+ 174us) server_negotiation.cc:407] Connection header received
0927 23:04:57.645856 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:57.645861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:57.645929 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:57.646033 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:57.646766 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.647582 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.649255 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.649629 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.649723 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:57.654576 (+ 4853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:40691: BlockingRecv error: recv got EOF from 127.3.240.195:40691 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Sep 27 23:04:57 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:57 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:57.842114 13265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:57.833143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:52327)
0927 23:04:57.833657 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:57.833672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:57.833821 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:57.834171 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:57.834178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:57.834556 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:57.834715 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:57.834723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.835913 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.835917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:57.837071 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:57.837082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.837847 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.837852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:57.838066 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:57.838723 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:57.838739 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:57.841882 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":275,"threads_started":1}
W20260927 23:04:57.842275 13263 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:57.833208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52327 (local address 127.3.240.254:38567)
0927 23:04:57.833428 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:57.833437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:57.833760 (+ 323us) server_negotiation.cc:407] Connection header received
0927 23:04:57.833948 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:57.833958 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:57.834030 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:57.834132 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:57.834870 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.835754 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.837199 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:57.837706 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:57.837838 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:57.842064 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:52327: BlockingRecv error: recv got EOF from 127.3.240.194:52327 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 27 23:04:58 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:58 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:58.518071 13266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:58.508965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:37333)
0927 23:04:58.509441 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:58.509457 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:58.509553 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:58.509998 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:58.510002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:58.510509 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:58.510734 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:58.510745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.511801 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.511807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:58.513245 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:58.513259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.513865 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.513869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:58.513988 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:58.514483 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:58.514499 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:58.517757 (+ 3258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
W20260927 23:04:58.518312 13267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:58.509066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:37333 (local address 127.3.240.254:38567)
0927 23:04:58.509723 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:58.509727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:58.509742 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:04:58.509781 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:58.509784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:58.509847 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:58.509944 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:58.510862 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.511672 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.513363 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.513747 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.513835 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:58.517945 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:37333: BlockingRecv error: recv got EOF from 127.3.240.193:37333 (error 108)
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":484,"threads_started":1}
Sep 27 23:04:58 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:58 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:58.664074 13268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:58.655569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:60337)
0927 23:04:58.656106 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:58.656121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:58.656230 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:58.656513 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:58.656516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:58.656851 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:58.657008 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:58.657015 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.657902 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.657905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:58.658796 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:58.658806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.659647 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.659651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:58.659866 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:58.660477 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:58.660493 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:58.663718 (+ 3225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":321,"threads_started":1}
W20260927 23:04:58.664121 13267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:58.655631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:60337 (local address 127.3.240.254:38567)
0927 23:04:58.655862 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:58.655867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:58.656190 (+ 323us) server_negotiation.cc:407] Connection header received
0927 23:04:58.656334 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:58.656337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:58.656395 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:58.656477 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:58.657123 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.657778 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.658939 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.659512 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.659652 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:58.663901 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:60337: BlockingRecv error: recv got EOF from 127.3.240.195:60337 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 27 23:04:58 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:58 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:58.851281 13269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:58.842925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:37269)
0927 23:04:58.843374 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:58.843396 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:58.843496 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:58.843771 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:58.843774 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:58.844129 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:58.844260 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:58.844267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.845160 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.845164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:58.846715 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:58.846726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.847415 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.847419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:58.847527 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:58.848030 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:58.848045 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:58.851082 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":266,"threads_started":1}
W20260927 23:04:58.851413 13267 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:58.843161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:37269 (local address 127.3.240.254:38567)
0927 23:04:58.843352 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:58.843359 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:58.843469 (+ 110us) server_negotiation.cc:407] Connection header received
0927 23:04:58.843586 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:58.843589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:58.843652 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:58.843727 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:58.844372 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.845029 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.846829 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:58.847280 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:58.847376 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:58.851218 (+ 3842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:37269: BlockingRecv error: recv got EOF from 127.3.240.194:37269 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 27 23:04:59 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:59 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:59.530553 13270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:59.518940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:34533)
0927 23:04:59.519477 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:59.519493 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:59.519608 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:59.519890 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:59.519892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:59.520302 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:59.520497 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:59.520506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.521447 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.521450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:59.522293 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:59.522302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.522909 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.522913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:59.523085 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:59.526897 (+ 3812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:59.526920 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:59.530220 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":319,"threads_started":1}
W20260927 23:04:59.530664 13271 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:59.519047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:34533 (local address 127.3.240.254:38567)
0927 23:04:59.519494 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:59.519498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:59.519542 (+ 44us) server_negotiation.cc:407] Connection header received
0927 23:04:59.519693 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:59.519696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:59.519750 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:59.519833 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:59.520614 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.521333 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.522410 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.522776 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.522858 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:59.530406 (+ 7548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:34533: BlockingRecv error: recv got EOF from 127.3.240.193:34533 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
Sep 27 23:04:59 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:59 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:59.673233 13272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:59.664918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:45099)
0927 23:04:59.665419 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:59.665435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:59.665586 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:59.665920 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:59.665927 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:59.666316 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:59.666454 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:59.666462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.667581 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.667587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:59.668532 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:59.668542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.669179 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.669184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:59.669351 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:59.669866 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:59.669898 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:59.672926 (+ 3028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
W20260927 23:04:59.673309 13271 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:59.665079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:45099 (local address 127.3.240.254:38567)
0927 23:04:59.665309 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:59.665315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:59.665518 (+ 203us) server_negotiation.cc:407] Connection header received
0927 23:04:59.665701 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:59.665706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:59.665772 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:59.665871 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:59.666602 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.667441 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.668665 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.669047 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.669169 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:59.673094 (+ 3925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:45099: BlockingRecv error: recv got EOF from 127.3.240.195:45099 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 27 23:04:59 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:04:59 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:04:59.866838 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:04:59.852375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:59257)
0927 23:04:59.852814 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 23:04:59.852827 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:04:59.852965 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:04:59.853326 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:04:59.853329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:04:59.853697 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:04:59.853840 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:59.853847 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.854715 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.854718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:59.855509 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:04:59.855518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.856043 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.856046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:04:59.856144 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:59.863099 (+ 6955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:04:59.863115 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:04:59.866511 (+ 3396us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":225,"threads_started":1}
W20260927 23:04:59.866921 13271 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:04:59.852456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:59257 (local address 127.3.240.254:38567)
0927 23:04:59.852640 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 23:04:59.852645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:04:59.852904 (+ 259us) server_negotiation.cc:407] Connection header received
0927 23:04:59.853118 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:04:59.853125 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:04:59.853217 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:04:59.853285 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:04:59.853966 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.854617 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.855604 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:04:59.855936 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:04:59.856014 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:04:59.866683 (+ 10669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:59257: BlockingRecv error: recv got EOF from 127.3.240.194:59257 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 27 23:05:00 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:00 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:00.540263 13274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:00.531460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:59723)
0927 23:05:00.532238 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:00.532253 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:00.532401 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:00.532683 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:00.532686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:00.533077 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:00.533236 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:00.533244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.534154 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.534157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:00.535132 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:00.535143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.536019 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.536028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:00.536191 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:00.536832 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:00.536849 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:00.540035 (+ 3186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":498,"threads_started":1}
W20260927 23:05:00.540428 13275 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:00.531536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:59723 (local address 127.3.240.254:38567)
0927 23:05:00.531951 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:00.531955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:00.532342 (+ 387us) server_negotiation.cc:407] Connection header received
0927 23:05:00.532502 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:00.532505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:00.532565 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:00.532649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:00.533353 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.534041 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.535304 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.535850 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.535943 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:00.540210 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:59723: BlockingRecv error: recv got EOF from 127.3.240.193:59723 (error 108)
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
Sep 27 23:05:00 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:00 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:00.685214 13276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:00.674176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:42787)
0927 23:05:00.674658 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:00.674673 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:00.674785 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:00.675045 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:00.675048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:00.675433 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:00.675586 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:00.675593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.676982 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.676986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:00.678341 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:00.678355 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.679483 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.679489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:00.679686 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:00.680402 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:00.680425 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:00.684979 (+ 4554us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":281,"threads_started":1}
W20260927 23:05:00.685527 13275 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:00.674304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:42787 (local address 127.3.240.254:38567)
0927 23:05:00.674502 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:00.674506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:00.674753 (+ 247us) server_negotiation.cc:407] Connection header received
0927 23:05:00.674876 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:00.674879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:00.674933 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:00.675028 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:00.675711 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.676791 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.678517 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.679258 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.679409 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:00.685152 (+ 5743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:42787: BlockingRecv error: recv got EOF from 127.3.240.195:42787 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 23:05:00 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:00 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:00.876636 13277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:00.867744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:57125)
0927 23:05:00.868308 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:00.868323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:00.868490 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:00.868838 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:00.868845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:00.869238 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:00.869396 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:00.869404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.870387 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.870395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:00.871404 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:00.871426 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.872111 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.872117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:00.872270 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:00.872942 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:00.872959 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:00.876405 (+ 3446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":286,"threads_started":1}
W20260927 23:05:00.876821 13275 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:00.867822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:57125 (local address 127.3.240.254:38567)
0927 23:05:00.868060 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:00.868069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:00.868426 (+ 357us) server_negotiation.cc:407] Connection header received
0927 23:05:00.868610 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:00.868614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:00.868690 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:00.868791 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:00.869532 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.870254 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.871544 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:00.871960 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:00.872085 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:00.876577 (+ 4492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:57125: BlockingRecv error: recv got EOF from 127.3.240.194:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 23:05:01 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:01 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:01.549407 13278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:01.541225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:46123)
0927 23:05:01.541715 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:01.541731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:01.541837 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:01.542275 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:01.542278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:01.542669 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:01.542848 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:01.542857 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.543884 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.543888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:01.544788 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:01.544799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.545418 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.545423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:01.545527 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:01.546080 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:01.546096 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:01.549183 (+ 3087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":251,"threads_started":1}
W20260927 23:05:01.549583 13279 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:01.541316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:46123 (local address 127.3.240.254:38567)
0927 23:05:01.542041 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:01.542045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:01.542059 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:05:01.542098 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:01.542101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:01.542154 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:01.542236 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:01.542969 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.543729 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.544908 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.545277 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.545360 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:01.549359 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:46123: BlockingRecv error: recv got EOF from 127.3.240.193:46123 (error 108)
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":504,"threads_started":1}
Sep 27 23:05:01 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:01 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:01.694571 13280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:01.686030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:41405)
0927 23:05:01.686574 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:01.686589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:01.686707 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:01.686985 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:01.687010 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:01.687392 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:01.687561 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:01.687569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.688513 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.688521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:01.689461 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:01.689472 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.690136 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.690140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:01.690252 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:01.690859 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:01.690910 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:01.694341 (+ 3431us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":257,"threads_started":1}
W20260927 23:05:01.694697 13279 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:01.686305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:41405 (local address 127.3.240.254:38567)
0927 23:05:01.686533 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:01.686539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:01.686660 (+ 121us) server_negotiation.cc:407] Connection header received
0927 23:05:01.686802 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:01.686806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:01.686865 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:01.686949 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:01.687687 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.688375 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.689587 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.690003 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.690099 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:01.694514 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:41405: BlockingRecv error: recv got EOF from 127.3.240.195:41405 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 27 23:05:01 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:01 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:01.886552 13281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:01.877573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:42113)
0927 23:05:01.878071 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:01.878086 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:01.878193 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:01.878503 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:01.878506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:01.878872 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:01.879064 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:01.879072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.880207 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.880210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:01.881159 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:01.881170 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.881894 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.881899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:01.882005 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:01.882720 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:01.882768 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:01.886315 (+ 3547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":285,"threads_started":1}
W20260927 23:05:01.886749 13279 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:01.877662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42113 (local address 127.3.240.254:38567)
0927 23:05:01.877941 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:01.877949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:01.878169 (+ 220us) server_negotiation.cc:407] Connection header received
0927 23:05:01.878295 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:01.878298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:01.878379 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:01.878470 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:01.879184 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.880068 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.881300 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:01.881738 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:01.881830 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:01.886504 (+ 4674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:42113: BlockingRecv error: recv got EOF from 127.3.240.194:42113 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 27 23:05:02 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:02 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:02.559299 13282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:02.550382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:42499)
0927 23:05:02.551160 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:02.551176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:02.551329 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:02.551668 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:02.551673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:02.552117 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:02.552262 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:02.552270 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.553280 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.553286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:02.554209 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:02.554219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.554908 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.554913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:02.555126 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:02.555734 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:02.555750 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:02.559076 (+ 3326us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":406,"threads_started":1}
W20260927 23:05:02.559527 13283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:02.550456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:42499 (local address 127.3.240.254:38567)
0927 23:05:02.550980 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:02.551008 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:02.551262 (+ 254us) server_negotiation.cc:407] Connection header received
0927 23:05:02.551455 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:02.551461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:02.551529 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:02.551637 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:02.552410 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.553142 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.554344 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.554759 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.554899 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:02.559244 (+ 4345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:42499: BlockingRecv error: recv got EOF from 127.3.240.193:42499 (error 108)
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":225,"threads_started":1}
Sep 27 23:05:02 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:02 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:02.704208 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:02.695500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:46009)
0927 23:05:02.696034 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:02.696050 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:02.696169 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:02.696466 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:02.696468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:02.696833 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:02.697049 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:02.697067 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.698039 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.698042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:02.698968 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:02.698978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.699637 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.699641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:02.699749 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:02.700306 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:02.700322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:02.703890 (+ 3568us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":273,"threads_started":1}
W20260927 23:05:02.704324 13283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:02.695591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:46009 (local address 127.3.240.254:38567)
0927 23:05:02.695745 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:02.695750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:02.696158 (+ 408us) server_negotiation.cc:407] Connection header received
0927 23:05:02.696249 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:02.696253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:02.696345 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:02.696431 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:02.697181 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.697926 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.699097 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.699498 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.699586 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:02.704044 (+ 4458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:46009: BlockingRecv error: recv got EOF from 127.3.240.195:46009 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 27 23:05:02 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:02 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:02.895941 13285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:02.887473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:46777)
0927 23:05:02.887966 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:02.887980 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:02.888109 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:02.888405 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:02.888408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:02.888752 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:02.888883 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:02.888890 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.889774 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.889780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:02.890968 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:02.890979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.891829 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.891834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:02.892010 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:02.892533 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:02.892548 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:02.895708 (+ 3160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":276,"threads_started":1}
W20260927 23:05:02.896126 13283 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:02.887507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:46777 (local address 127.3.240.254:38567)
0927 23:05:02.887682 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:02.887686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:02.888069 (+ 383us) server_negotiation.cc:407] Connection header received
0927 23:05:02.888207 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:02.888210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:02.888279 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:02.888358 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:02.888993 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.889651 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.891152 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:02.891677 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:02.891805 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:02.895891 (+ 4086us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:46777: BlockingRecv error: recv got EOF from 127.3.240.194:46777 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 27 23:05:03 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:03 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:03.569113 13286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:03.560264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:48631)
0927 23:05:03.560781 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:03.560798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:03.560914 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:03.561217 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:03.561220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:03.561607 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:03.561800 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:03.561813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.563103 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.563111 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:03.564128 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:03.564139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.564863 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.564869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:03.565050 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:03.565609 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:03.565626 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:03.568744 (+ 3118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":260,"threads_started":1}
W20260927 23:05:03.569185 13287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:03.560321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:48631 (local address 127.3.240.254:38567)
0927 23:05:03.560796 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:03.560801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:03.560873 (+ 72us) server_negotiation.cc:407] Connection header received
0927 23:05:03.561013 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:03.561016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:03.561088 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:03.561176 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:03.561980 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.562927 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.564281 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.564710 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.564844 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:03.568928 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:48631: BlockingRecv error: recv got EOF from 127.3.240.193:48631 (error 108)
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":271,"threads_started":1}
Sep 27 23:05:03 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:03 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:03.713965 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:03.705204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:44257)
0927 23:05:03.705726 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:03.705746 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:03.705894 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:03.706290 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:03.706296 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:03.706769 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:03.706920 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:03.706928 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.707951 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.707955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:03.708880 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:03.708890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.709534 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.709538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:03.709666 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:03.710226 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:03.710242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:03.713635 (+ 3393us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":304,"threads_started":1}
W20260927 23:05:03.714046 13287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:03.705265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:44257 (local address 127.3.240.254:38567)
0927 23:05:03.705484 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:03.705492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:03.705850 (+ 358us) server_negotiation.cc:407] Connection header received
0927 23:05:03.706018 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:03.706024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:03.706107 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:03.706205 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:03.707104 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.707825 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.709015 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.709410 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.709502 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:03.713806 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:44257: BlockingRecv error: recv got EOF from 127.3.240.195:44257 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 27 23:05:03 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:03 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:03.905812 13289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:03.896808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:51219)
0927 23:05:03.897321 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:03.897335 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:03.897496 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:03.897812 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:03.897817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:03.898178 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:03.898324 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:03.898332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.899388 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.899392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:03.900303 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:03.900314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.901004 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.901009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:03.901168 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:03.901842 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:03.901860 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:03.905573 (+ 3713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":311,"threads_started":1}
W20260927 23:05:03.905963 13287 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:03.896827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:51219 (local address 127.3.240.254:38567)
0927 23:05:03.897008 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:03.897015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:03.897429 (+ 414us) server_negotiation.cc:407] Connection header received
0927 23:05:03.897606 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:03.897611 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:03.897690 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:03.897785 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:03.898466 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.899238 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.900461 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:03.900852 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:03.900993 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:03.905742 (+ 4749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:51219: BlockingRecv error: recv got EOF from 127.3.240.194:51219 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 27 23:05:04 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:04 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:04.586777 13290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:04.570088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:55947)
0927 23:05:04.570582 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:04.570597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:04.570703 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:04.571163 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:04.571166 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:04.571554 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:04.571724 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:04.571732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.572870 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.572873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:04.573674 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:04.573682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.574340 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.574344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:04.574468 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:04.583122 (+ 8654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:04.583139 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:04.586497 (+ 3358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":268,"threads_started":1}
W20260927 23:05:04.586946 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:04.570314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:55947 (local address 127.3.240.254:38567)
0927 23:05:04.570873 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:04.570877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:04.570893 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:05:04.570937 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:04.570941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:04.570997 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:04.571113 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:04.571863 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.572725 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.573803 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.574215 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.574313 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:04.586708 (+ 12395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:55947: BlockingRecv error: recv got EOF from 127.3.240.193:55947 (error 108)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":312,"threads_started":1}
Sep 27 23:05:04 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:04 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:04.723523 13292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:04.714866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:48039)
0927 23:05:04.715382 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:04.715398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:04.715504 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:04.715913 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:04.715918 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:04.716289 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:04.716510 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:04.716519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.717730 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.717733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:04.718622 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:04.718632 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.719306 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.719309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:04.719417 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:04.720014 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:04.720030 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:04.723258 (+ 3228us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":278,"threads_started":1}
W20260927 23:05:04.723572 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:04.714968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:48039 (local address 127.3.240.254:38567)
0927 23:05:04.715144 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:04.715148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:04.715486 (+ 338us) server_negotiation.cc:407] Connection header received
0927 23:05:04.715609 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:04.715614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:04.715701 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:04.715876 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:04.716670 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.717604 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.718741 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.719193 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.719291 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:04.723415 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:48039: BlockingRecv error: recv got EOF from 127.3.240.195:48039 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 27 23:05:04 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:04 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:04.915407 13293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:04.906751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:52249)
0927 23:05:04.907242 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:04.907256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:04.907398 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:04.907728 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:04.907732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:04.908168 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:04.908327 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:04.908335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.909268 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.909276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:04.910228 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:04.910238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.910923 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.910929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:04.911122 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:04.911655 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:04.911671 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:04.915180 (+ 3509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":259,"threads_started":1}
W20260927 23:05:04.915628 13291 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:04.906825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52249 (local address 127.3.240.254:38567)
0927 23:05:04.907052 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:04.907059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:04.907333 (+ 274us) server_negotiation.cc:407] Connection header received
0927 23:05:04.907520 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:04.907525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:04.907601 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:04.907698 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:04.908460 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.909146 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.910364 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:04.910765 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:04.910891 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:04.915372 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:52249: BlockingRecv error: recv got EOF from 127.3.240.194:52249 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 27 23:05:05 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:05 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:05.597520 13294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:05.587817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:42317)
0927 23:05:05.588540 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:05.588556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:05.588655 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:05.589108 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:05.589111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:05.589481 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:05.589635 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:05.589643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.590881 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.590889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:05.592136 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:05.592148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.592790 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.592794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:05.592916 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:05.593514 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:05.593530 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:05.597270 (+ 3740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":503,"threads_started":1}
W20260927 23:05:05.597731 13295 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:05.587874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:42317 (local address 127.3.240.254:38567)
0927 23:05:05.588867 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:05.588871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:05.588884 (+ 13us) server_negotiation.cc:407] Connection header received
0927 23:05:05.588923 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:05.588926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:05.588983 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:05.589065 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:05.589753 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.590746 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.592250 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.592648 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.592735 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:05.597474 (+ 4739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:42317: BlockingRecv error: recv got EOF from 127.3.240.193:42317 (error 108)
Metrics: {"server-negotiator.queue_time_us":910,"thread_start_us":775,"threads_started":1}
Sep 27 23:05:05 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:05 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:05.732910 13296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:05.724485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:46969)
0927 23:05:05.724985 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:05.725000 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:05.725111 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:05.725493 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:05.725497 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:05.725868 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:05.726041 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:05.726048 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.726974 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.726978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:05.727932 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:05.727943 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.728583 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.728587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:05.728697 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:05.729233 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:05.729249 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:05.732691 (+ 3442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":273,"threads_started":1}
W20260927 23:05:05.733071 13295 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:05.724580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:46969 (local address 127.3.240.254:38567)
0927 23:05:05.724805 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:05.724812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:05.725080 (+ 268us) server_negotiation.cc:407] Connection header received
0927 23:05:05.725296 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:05.725300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:05.725364 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:05.725447 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:05.726164 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.726847 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.728056 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.728449 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.728555 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:05.732869 (+ 4314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:46969: BlockingRecv error: recv got EOF from 127.3.240.195:46969 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 27 23:05:05 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:05 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:05.924611 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:05.916223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:33625)
0927 23:05:05.916749 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:05.916763 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:05.916869 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:05.917139 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:05.917142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:05.917494 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:05.917641 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:05.917648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.918897 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.918903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:05.919920 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:05.919931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.920578 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.920582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:05.920698 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:05.921231 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:05.921247 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:05.924318 (+ 3071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":299,"threads_started":1}
W20260927 23:05:05.924962 13295 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:05.916254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:33625 (local address 127.3.240.254:38567)
0927 23:05:05.916402 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:05.916406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:05.916829 (+ 423us) server_negotiation.cc:407] Connection header received
0927 23:05:05.916961 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:05.916964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:05.917031 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:05.917103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:05.917771 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.918740 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.920036 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:05.920433 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:05.920522 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:05.924507 (+ 3985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:33625: BlockingRecv error: recv got EOF from 127.3.240.194:33625 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 23:05:06 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:06 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:06.606619 13298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:06.598406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:44219)
0927 23:05:06.598825 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:06.598841 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:06.598941 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:06.599473 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:06.599476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:06.599848 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:06.599986 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:06.599995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.600925 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.600929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:06.601812 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:06.601822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.602435 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.602439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:06.602579 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:06.603191 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:06.603207 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:06.606397 (+ 3190us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":228,"threads_started":1}
W20260927 23:05:06.606805 13299 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:06.598444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:44219 (local address 127.3.240.254:38567)
0927 23:05:06.599228 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:06.599232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:06.599245 (+ 13us) server_negotiation.cc:407] Connection header received
0927 23:05:06.599283 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:06.599286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:06.599337 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:06.599430 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:06.600109 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.600807 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.601927 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.602317 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.602408 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:06.606574 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:44219: BlockingRecv error: recv got EOF from 127.3.240.193:44219 (error 108)
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":584,"threads_started":1}
Sep 27 23:05:06 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:06 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:06.743431 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:06.733781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:35491)
0927 23:05:06.734353 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:06.734369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:06.734479 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:06.734751 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:06.734754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:06.735139 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:06.735296 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:06.735304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.736403 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.736406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:06.737866 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:06.737882 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.738700 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.738708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:06.738901 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:06.739515 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:06.739542 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:06.743205 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":321,"threads_started":1}
W20260927 23:05:06.743793 13299 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:06.733860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:35491 (local address 127.3.240.254:38567)
0927 23:05:06.734068 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:06.734073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:06.734444 (+ 371us) server_negotiation.cc:407] Connection header received
0927 23:05:06.734560 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:06.734563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:06.734624 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:06.734703 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:06.735451 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.736268 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.738005 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.738523 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.738622 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:06.743422 (+ 4800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:35491: BlockingRecv error: recv got EOF from 127.3.240.195:35491 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 27 23:05:06 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:06 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:06.934192 13301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:06.925565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:40583)
0927 23:05:06.926054 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:06.926071 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:06.926179 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:06.926471 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:06.926475 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:06.926845 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:06.927061 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:06.927071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.928484 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.928489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:06.929490 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:06.929502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.930113 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.930116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:06.930226 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:06.930812 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:06.930828 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:06.933969 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":280,"threads_started":1}
W20260927 23:05:06.934378 13299 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:06.925645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:40583 (local address 127.3.240.254:38567)
0927 23:05:06.925814 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:06.925819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:06.926140 (+ 321us) server_negotiation.cc:407] Connection header received
0927 23:05:06.926270 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:06.926273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:06.926346 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:06.926429 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:06.927205 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.928328 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.929618 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:06.930001 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:06.930091 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:06.934147 (+ 4056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:40583: BlockingRecv error: recv got EOF from 127.3.240.194:40583 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 23:05:07 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:07 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:07.616375 13302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:07.607713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.193:48975)
0927 23:05:07.608160 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:07.608176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:07.608267 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:07.608711 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:07.608714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:07.609181 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:07.609382 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:07.609391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.610560 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.610566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:07.611584 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:07.611596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.612336 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.612343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:07.612508 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:07.613059 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:07.613074 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:07.616023 (+ 2949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":233,"threads_started":1}
W20260927 23:05:07.616436 13303 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:07.607802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:48975 (local address 127.3.240.254:38567)
0927 23:05:07.608473 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:07.608477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:07.608491 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:05:07.608528 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:07.608530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:07.608584 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:07.608669 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:07.609510 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.610429 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.611717 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.612201 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.612312 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:07.616193 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:48975: BlockingRecv error: recv got EOF from 127.3.240.193:48975 (error 108)
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":487,"threads_started":1}
Sep 27 23:05:07 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:07 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:07.753118 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:07.744398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.195:48901)
0927 23:05:07.744882 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:07.744897 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:07.745022 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:07.745347 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:07.745350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:07.745737 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:07.745924 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:07.745933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.746956 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.746960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:07.747896 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:07.747907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.748531 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.748535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:07.748646 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:07.749258 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:07.749281 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:07.752918 (+ 3637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":263,"threads_started":1}
W20260927 23:05:07.753315 13303 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:07.744453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:48901 (local address 127.3.240.254:38567)
0927 23:05:07.744740 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:07.744746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:07.745026 (+ 280us) server_negotiation.cc:407] Connection header received
0927 23:05:07.745130 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:07.745135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:07.745212 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:07.745319 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:07.746064 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.746840 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.748011 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.748419 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.748506 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:07.753081 (+ 4575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:48901: BlockingRecv error: recv got EOF from 127.3.240.195:48901 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
Sep 27 23:05:07 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Sep 27 23:05:07 dist-test-slave-lmfp krb5kdc[12244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260927 23:05:07.943902 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:07.935441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38567 (local address 127.3.240.194:42201)
0927 23:05:07.936164 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:07.936182 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:07.936306 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:07.936695 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:07.936700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:07.937122 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:07.937304 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:07.937312 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.938257 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.938262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:07.939249 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:07.939260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.939912 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.939918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:07.940101 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:07.940675 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:07.940690 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:07.943660 (+ 2970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:38567: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":276,"threads_started":1}
W20260927 23:05:07.944036 13303 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:07.935702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42201 (local address 127.3.240.254:38567)
0927 23:05:07.935904 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:07.935911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:07.936265 (+ 354us) server_negotiation.cc:407] Connection header received
0927 23:05:07.936460 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:07.936465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:07.936546 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:07.936647 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:07.937434 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.938128 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.939382 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:07.939777 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:07.939904 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:07.943825 (+ 3921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:42201: BlockingRecv error: recv got EOF from 127.3.240.194:42201 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:05:08.526650 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12831
I20260927 23:05:08.538388 13307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:08.527752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.3.240.193:39691)
0927 23:05:08.528420 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:08.528427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:08.528454 (+ 27us) server_negotiation.cc:407] Connection header received
0927 23:05:08.528511 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:08.528517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:08.528759 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:08.528915 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:08.530219 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:08.531280 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:08.532456 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:08.532996 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:08.533091 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:08.535321 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:08.535328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:08.535341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:08.535404 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:08.537916 (+ 2512us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 23:05:08.538184 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43842: Request ticket server kudu/127.3.240.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":306,"threads_started":1}
W20260927 23:05:08.538513 13307 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43842: Request ticket server kudu/127.3.240.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 23:05:08.538611 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12831: Not authorized: Client connection negotiation failed: client connection to 127.3.240.193:39691: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 23:05:08.538846 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 12831
I20260927 23:05:08.555675 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12982
I20260927 23:05:08.567911 13308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:08.556646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.3.240.194:33917)
0927 23:05:08.557183 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:08.557190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:08.557213 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:05:08.557278 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:08.557285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:08.557528 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:08.557803 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:08.559548 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:08.560539 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:08.561731 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:08.562304 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:08.562430 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:08.564803 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:08.564811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:08.564824 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:08.564874 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:08.567427 (+ 2553us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 23:05:08.567664 (+ 237us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58026: Request ticket server kudu/127.3.240.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":254,"threads_started":1}
W20260927 23:05:08.568024 13308 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58026: Request ticket server kudu/127.3.240.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 23:05:08.567988 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12982: Not authorized: Client connection negotiation failed: client connection to 127.3.240.194:33917: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 23:05:08.568140 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 12982
I20260927 23:05:08.583714 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 13115
I20260927 23:05:08.595913 13309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:08.584703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.3.240.195:39321)
0927 23:05:08.585376 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:08.585385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:08.585415 (+ 30us) server_negotiation.cc:407] Connection header received
0927 23:05:08.585485 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:08.585494 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:08.585784 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:08.585982 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:08.587174 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:08.588636 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:08.589667 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:08.590272 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:08.590464 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:08.592886 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:08.592894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:08.592905 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:08.592944 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:08.595368 (+ 2424us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0927 23:05:08.595626 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36690: Request ticket server kudu/127.3.240.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":301,"threads_started":1}
W20260927 23:05:08.596009 13309 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36690: Request ticket server kudu/127.3.240.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260927 23:05:08.596145 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 13115: Not authorized: Client connection negotiation failed: client connection to 127.3.240.195:39321: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260927 23:05:08.596298 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 13115
I20260927 23:05:08.612586 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 12762
I20260927 23:05:08.613749 12824 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:08.725090 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 12762
2026-09-27T23:05:08Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21210 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-27T23:05:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:05:08Z Disabled control of system clock
I20260927 23:05:08.796554 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:33227
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:33227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260927 23:05:09.099956 13314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:09.100214 13314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:09.100343 13314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:09.110407 13314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:05:09.110553 13314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:09.110642 13314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:09.110728 13314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:05:09.110795 13314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:05:09.121726 13314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:33227
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:33227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:09.123131 13314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:09.125427 13314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:09.137185 13319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:09.137253 13320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:09.137253 13322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:09.137524 13314 server_base.cc:1034] running on GCE node
I20260927 23:05:09.138518 13314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:09.139974 13314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:09.141209 13314 hybrid_clock.cc:648] HybridClock initialized: now 1790550309141178 us; error 43 us; skew 500 ppm
I20260927 23:05:09.141533 13314 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:05:09.143678 13314 webserver.cc:460] Webserver started at http://127.3.240.254:34685/ using document root <none> and password file <none>
I20260927 23:05:09.144356 13314 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:09.144481 13314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:09.144788 13314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:09.147579 13314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89"
format_stamp: "Formatted at 2026-09-27 23:05:09 on dist-test-slave-lmfp"
server_key: "72b72defc11341872e7350e921c652d0"
server_key_iv: "06c1313554c74d27a3769b48175440b0"
server_key_version: "encryptionkey@0"
I20260927 23:05:09.148336 13314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89"
format_stamp: "Formatted at 2026-09-27 23:05:09 on dist-test-slave-lmfp"
server_key: "72b72defc11341872e7350e921c652d0"
server_key_iv: "06c1313554c74d27a3769b48175440b0"
server_key_version: "encryptionkey@0"
I20260927 23:05:09.152951 13314 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 23:05:09.156072 13328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:09.157269 13314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:05:09.157433 13314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89"
format_stamp: "Formatted at 2026-09-27 23:05:09 on dist-test-slave-lmfp"
server_key: "72b72defc11341872e7350e921c652d0"
server_key_iv: "06c1313554c74d27a3769b48175440b0"
server_key_version: "encryptionkey@0"
I20260927 23:05:09.157655 13314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:09.184422 13314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:09.185221 13314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:09.185506 13314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:09.202809 13314 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:33227
I20260927 23:05:09.202816 13379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:33227 every 8 connection(s)
I20260927 23:05:09.204167 13314 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:05:09.205353 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 13314
I20260927 23:05:09.205533 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:05:09.206357 4035 external_mini_cluster.cc:1428] Setting key 589d07c5eb396bad04597ac30bec78fa
I20260927 23:05:09.208981 13380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:05:09.218623 13383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:09.208340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.3.240.254:33227)
0927 23:05:09.209277 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:09.209288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:09.209325 (+ 37us) server_negotiation.cc:407] Connection header received
0927 23:05:09.210802 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:09.210821 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:09.211363 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:09.211735 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:09.212258 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:09.213396 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:09.214587 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:09.215236 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:09.215434 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:09.216529 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:09.216538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:09.216550 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:09.217090 (+ 540us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:09.217160 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:09.217183 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:09.217303 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:09.217522 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 23:05:09.217788 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":303,"threads_started":1}
I20260927 23:05:09.218703 13380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89: Bootstrap starting.
I20260927 23:05:09.221716 13380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89: Neither blocks nor log segments found. Creating new log.
I20260927 23:05:09.222716 13380 log.cc:824] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89: Log is configured to *not* fsync() on all Append() calls
I20260927 23:05:09.225157 13380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89: No bootstrap required, opened a new log
I20260927 23:05:09.231736 13380 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33227 } }
I20260927 23:05:09.232045 13380 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:05:09.232127 13380 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ce8e26c0da4ee59b1656e43e9b3c89, State: Initialized, Role: FOLLOWER
I20260927 23:05:09.232623 13380 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33227 } }
I20260927 23:05:09.232803 13380 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:05:09.232898 13380 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:05:09.233011 13380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:05:09.234922 13380 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33227 } }
I20260927 23:05:09.235430 13380 leader_election.cc:302] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ce8e26c0da4ee59b1656e43e9b3c89; no voters:
I20260927 23:05:09.235972 13380 leader_election.cc:288] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:05:09.236145 13385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:05:09.236872 13385 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [term 1 LEADER]: Becoming Leader. State: Replica: b4ce8e26c0da4ee59b1656e43e9b3c89, State: Running, Role: LEADER
I20260927 23:05:09.237424 13385 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33227 } }
I20260927 23:05:09.238113 13380 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:05:09.240170 13386 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33227 } } }
I20260927 23:05:09.240525 13386 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [sys.catalog]: This master's current role is: LEADER
I20260927 23:05:09.240332 13387 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ce8e26c0da4ee59b1656e43e9b3c89. Latest consensus state: current_term: 1 leader_uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ce8e26c0da4ee59b1656e43e9b3c89" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33227 } } }
I20260927 23:05:09.240710 13387 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89 [sys.catalog]: This master's current role is: LEADER
I20260927 23:05:09.241390 13393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:05:09.246742 13393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:05:09.255208 13393 catalog_manager.cc:1347] Generated new cluster ID: a86d2479de244f3d92a1b59c1e45426f
I20260927 23:05:09.255304 13393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:05:09.279721 13393 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:05:09.280920 13393 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:05:09.290385 13393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ce8e26c0da4ee59b1656e43e9b3c89: Generated new TSK 0
I20260927 23:05:09.291174 13393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:05:09.304769 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33227
--builtin_ntp_servers=127.3.240.212:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260927 23:05:09.638266 13404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:09.638500 13404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:09.638655 13404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:09.645814 13404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:09.645953 13404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:09.646088 13404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:05:09.654627 13404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:09.655972 13404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:09.657624 13404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:09.669997 13412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:09.670045 13410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:09.670223 13409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:09.670796 13404 server_base.cc:1034] running on GCE node
I20260927 23:05:09.671360 13404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:09.672181 13404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:09.673377 13404 hybrid_clock.cc:648] HybridClock initialized: now 1790550309673332 us; error 32 us; skew 500 ppm
I20260927 23:05:09.673668 13404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:05:09.679309 13404 webserver.cc:460] Webserver started at http://127.3.240.193:41649/ using document root <none> and password file <none>
I20260927 23:05:09.679991 13404 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:09.680099 13404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:09.680446 13404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:09.683215 13404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "4d30dbf69c9c46708077ed45b39af3be"
format_stamp: "Formatted at 2026-09-27 23:05:09 on dist-test-slave-lmfp"
server_key: "6e43aee75bb6c03946760add6dee9d59"
server_key_iv: "e2b9222a7b3689d89bb3391d16138f92"
server_key_version: "encryptionkey@0"
I20260927 23:05:09.684007 13404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "4d30dbf69c9c46708077ed45b39af3be"
format_stamp: "Formatted at 2026-09-27 23:05:09 on dist-test-slave-lmfp"
server_key: "6e43aee75bb6c03946760add6dee9d59"
server_key_iv: "e2b9222a7b3689d89bb3391d16138f92"
server_key_version: "encryptionkey@0"
I20260927 23:05:09.688895 13404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:05:09.692139 13418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:09.693401 13404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:05:09.693569 13404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "4d30dbf69c9c46708077ed45b39af3be"
format_stamp: "Formatted at 2026-09-27 23:05:09 on dist-test-slave-lmfp"
server_key: "6e43aee75bb6c03946760add6dee9d59"
server_key_iv: "e2b9222a7b3689d89bb3391d16138f92"
server_key_version: "encryptionkey@0"
I20260927 23:05:09.693765 13404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:09.707576 13404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:09.708305 13404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:09.708593 13404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:09.709441 13404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:09.710719 13404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:09.710836 13404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:09.710963 13404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:09.711062 13404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260927 23:05:09.743137 13404 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:34965
I20260927 23:05:09.743180 13530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:34965 every 8 connection(s)
I20260927 23:05:09.744329 13404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:05:09.747767 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 13404
I20260927 23:05:09.747968 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:05:09.748553 4035 external_mini_cluster.cc:1428] Setting key 446984cd719cea136c5c20f747c4b773
I20260927 23:05:09.750972 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33227
--builtin_ntp_servers=127.3.240.212:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 23:05:09.756460 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:09.746623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:33917 (local address 127.3.240.254:33227)
0927 23:05:09.747049 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:09.747054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:09.748620 (+ 1566us) server_negotiation.cc:407] Connection header received
0927 23:05:09.750289 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:09.750294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:09.750386 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:09.750499 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:09.751646 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:09.752326 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:09.753829 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:09.754243 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:09.754378 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:09.755651 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:09.755657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:09.755662 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:09.755686 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:09.755720 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:09.755731 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:09.755819 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:09.756051 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 23:05:09.756142 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":195,"threads_started":1}
I20260927 23:05:09.757342 13534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:09.747289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:33227 (local address 127.3.240.193:33917)
0927 23:05:09.748442 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:09.748491 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:09.750039 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:09.750717 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:09.750726 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:09.750810 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:09.751486 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:09.751503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:09.752491 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:09.752495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:09.753689 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:09.753699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:09.754457 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:09.754463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:09.754671 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:09.755368 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:05:09.755370 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:09.755407 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:05:09.755473 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:09.755834 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:09.755865 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:09.755870 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 23:05:09.756023 (+ 153us) client_negotiation.cc:240] Negotiation successful
0927 23:05:09.756369 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":327,"threads_started":1}
I20260927 23:05:09.758790 13531 heartbeater.cc:347] Connected to a master server at 127.3.240.254:33227
I20260927 23:05:09.759112 13531 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:09.759792 13531 heartbeater.cc:511] Master 127.3.240.254:33227 requested a full tablet report, sending...
I20260927 23:05:09.761739 13345 ts_manager.cc:194] Registered new tserver with Master: 4d30dbf69c9c46708077ed45b39af3be (127.3.240.193:34965)
I20260927 23:05:09.763327 13345 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.193:33917
W20260927 23:05:10.075685 13535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:10.075971 13535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:10.076123 13535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:10.082978 13535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:10.083143 13535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:10.083309 13535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:05:10.091830 13535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:10.092975 13535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:10.094656 13535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:10.107270 13540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:10.107400 13541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:10.107380 13543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:10.107872 13535 server_base.cc:1034] running on GCE node
I20260927 23:05:10.108466 13535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:10.109242 13535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:10.110455 13535 hybrid_clock.cc:648] HybridClock initialized: now 1790550310110399 us; error 43 us; skew 500 ppm
I20260927 23:05:10.110731 13535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:05:10.112789 13535 webserver.cc:460] Webserver started at http://127.3.240.194:45845/ using document root <none> and password file <none>
I20260927 23:05:10.113413 13535 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:10.113517 13535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:10.113808 13535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:10.116369 13535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "f3d46c940b5e459b9d9199f135183ea9"
format_stamp: "Formatted at 2026-09-27 23:05:10 on dist-test-slave-lmfp"
server_key: "98108e0b264a31cbeaa2dbfdecb4d979"
server_key_iv: "e259e21134b6400d3b9d6ea0a1adc392"
server_key_version: "encryptionkey@0"
I20260927 23:05:10.117056 13535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "f3d46c940b5e459b9d9199f135183ea9"
format_stamp: "Formatted at 2026-09-27 23:05:10 on dist-test-slave-lmfp"
server_key: "98108e0b264a31cbeaa2dbfdecb4d979"
server_key_iv: "e259e21134b6400d3b9d6ea0a1adc392"
server_key_version: "encryptionkey@0"
I20260927 23:05:10.122298 13535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 23:05:10.125612 13549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:10.126879 13535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:05:10.127107 13535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "f3d46c940b5e459b9d9199f135183ea9"
format_stamp: "Formatted at 2026-09-27 23:05:10 on dist-test-slave-lmfp"
server_key: "98108e0b264a31cbeaa2dbfdecb4d979"
server_key_iv: "e259e21134b6400d3b9d6ea0a1adc392"
server_key_version: "encryptionkey@0"
I20260927 23:05:10.127329 13535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:10.142292 13535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:10.143219 13535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:10.143498 13535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:10.144379 13535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:10.145797 13535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:10.145927 13535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:10.146044 13535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:10.146128 13535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:10.177999 13535 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:33955
I20260927 23:05:10.178050 13661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:33955 every 8 connection(s)
I20260927 23:05:10.179272 13535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:05:10.181903 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 13535
I20260927 23:05:10.182065 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:05:10.182595 4035 external_mini_cluster.cc:1428] Setting key b23aa4210c601be1c088f1d7c69ef353
I20260927 23:05:10.185205 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33227
--builtin_ntp_servers=127.3.240.212:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260927 23:05:10.191959 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:10.181709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:35111 (local address 127.3.240.254:33227)
0927 23:05:10.181869 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:10.181874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:10.183624 (+ 1750us) server_negotiation.cc:407] Connection header received
0927 23:05:10.185403 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:10.185408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:10.185484 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:10.185601 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:10.186604 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.187380 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.188865 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.189319 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.189453 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:10.191065 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:10.191074 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:10.191080 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:10.191105 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:10.191148 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:10.191167 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:10.191271 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:10.191471 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 23:05:10.191581 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:05:10.192937 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:10.182417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:33227 (local address 127.3.240.194:35111)
0927 23:05:10.183487 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:10.183527 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:10.185135 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:10.185819 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:10.185827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:10.185906 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:10.186440 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:10.186454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.187519 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.187527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:10.188696 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:10.188706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.189554 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.189563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:10.189888 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:10.190618 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:05:10.190624 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:10.190674 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:05:10.190796 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:10.191271 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:10.191279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:10.191284 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 23:05:10.191460 (+ 176us) client_negotiation.cc:240] Negotiation successful
0927 23:05:10.191819 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":282,"threads_started":1}
I20260927 23:05:10.194346 13662 heartbeater.cc:347] Connected to a master server at 127.3.240.254:33227
I20260927 23:05:10.194628 13662 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:10.195477 13662 heartbeater.cc:511] Master 127.3.240.254:33227 requested a full tablet report, sending...
I20260927 23:05:10.196808 13345 ts_manager.cc:194] Registered new tserver with Master: f3d46c940b5e459b9d9199f135183ea9 (127.3.240.194:33955)
I20260927 23:05:10.197678 13345 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.194:35111
W20260927 23:05:10.498243 13665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:10.498510 13665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:10.498657 13665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:10.505565 13665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:10.505656 13665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:10.505838 13665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:05:10.514389 13665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:10.515539 13665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:10.517146 13665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:10.530054 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:10.530340 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:10.530092 13673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:10.530897 13665 server_base.cc:1034] running on GCE node
I20260927 23:05:10.531599 13665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:10.532589 13665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:10.533834 13665 hybrid_clock.cc:648] HybridClock initialized: now 1790550310533745 us; error 75 us; skew 500 ppm
I20260927 23:05:10.534162 13665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:05:10.536506 13665 webserver.cc:460] Webserver started at http://127.3.240.195:37201/ using document root <none> and password file <none>
I20260927 23:05:10.537195 13665 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:10.537344 13665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:10.537669 13665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:10.540438 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "e556c04d8c9e4efeb9f1a3ea46a13bf5"
format_stamp: "Formatted at 2026-09-27 23:05:10 on dist-test-slave-lmfp"
server_key: "2fe4b4d92b1558134d6e35cd7415a551"
server_key_iv: "ca471f0039bc1308972ef394d959df9c"
server_key_version: "encryptionkey@0"
I20260927 23:05:10.541278 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "e556c04d8c9e4efeb9f1a3ea46a13bf5"
format_stamp: "Formatted at 2026-09-27 23:05:10 on dist-test-slave-lmfp"
server_key: "2fe4b4d92b1558134d6e35cd7415a551"
server_key_iv: "ca471f0039bc1308972ef394d959df9c"
server_key_version: "encryptionkey@0"
I20260927 23:05:10.546417 13665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 23:05:10.549499 13679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:10.550680 13665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:05:10.550848 13665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "e556c04d8c9e4efeb9f1a3ea46a13bf5"
format_stamp: "Formatted at 2026-09-27 23:05:10 on dist-test-slave-lmfp"
server_key: "2fe4b4d92b1558134d6e35cd7415a551"
server_key_iv: "ca471f0039bc1308972ef394d959df9c"
server_key_version: "encryptionkey@0"
I20260927 23:05:10.551134 13665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:10.568885 13665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:10.569654 13665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:10.569944 13665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:10.570863 13665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:10.572288 13665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:10.572402 13665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:10.572539 13665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:10.572626 13665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:10.604112 13665 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42111
I20260927 23:05:10.604162 13791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42111 every 8 connection(s)
I20260927 23:05:10.605348 13665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:05:10.607327 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 13665
I20260927 23:05:10.607482 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:05:10.607950 4035 external_mini_cluster.cc:1428] Setting key 05ce9ef3013f723967441fe75e3f8f7b
I20260927 23:05:10.617036 13533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:10.607678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:43969 (local address 127.3.240.254:33227)
0927 23:05:10.607893 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:10.607899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:10.609593 (+ 1694us) server_negotiation.cc:407] Connection header received
0927 23:05:10.611667 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:10.611670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:10.611728 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:10.611805 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:10.612706 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.613367 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.614787 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.615246 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.615351 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:10.616461 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:10.616466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:10.616469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:10.616484 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:10.616511 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:10.616521 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:10.616583 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:10.616755 (+ 172us) server_negotiation.cc:299] Negotiation successful
0927 23:05:10.616815 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:05:10.617937 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:10.608293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:33227 (local address 127.3.240.195:43969)
0927 23:05:10.609408 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:10.609462 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:10.611452 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:10.612000 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:10.612007 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:10.612084 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:10.612588 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:10.612602 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.613474 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.613477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:10.614672 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:10.614686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.615436 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.615441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:10.615613 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:10.616216 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:05:10.616219 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:10.616252 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:05:10.616314 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:10.616596 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:10.616603 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:10.616608 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 23:05:10.616731 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 23:05:10.616985 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":343,"threads_started":1}
I20260927 23:05:10.619083 13792 heartbeater.cc:347] Connected to a master server at 127.3.240.254:33227
I20260927 23:05:10.619331 13792 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:10.619994 13792 heartbeater.cc:511] Master 127.3.240.254:33227 requested a full tablet report, sending...
I20260927 23:05:10.621325 13345 ts_manager.cc:194] Registered new tserver with Master: e556c04d8c9e4efeb9f1a3ea46a13bf5 (127.3.240.195:42111)
I20260927 23:05:10.622219 13345 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.195:43969
I20260927 23:05:10.623346 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 23:05:10.629101 13533 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:10.627528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.3.240.254:33227)
0927 23:05:10.627655 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:10.627659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:10.628200 (+ 541us) server_negotiation.cc:407] Connection header received
0927 23:05:10.628368 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:10.628374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:10.628434 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:10.628522 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:10.628872 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35390: BlockingRecv error: recv got EOF from 127.0.0.1:35390 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 23:05:10.629794 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 13404
I20260927 23:05:10.636919 13801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:10.630812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36398 (local address 127.3.240.193:34965)
0927 23:05:10.631294 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:10.631300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:10.631320 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:05:10.631373 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:10.631378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:10.631537 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:10.631686 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:10.632121 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.633048 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.634140 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.634608 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.634728 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:10.635711 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:10.635718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:10.635727 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:10.636221 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:10.636258 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:10.636275 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:10.636429 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:10.636525 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 23:05:10.636632 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260927 23:05:10.637874 13526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:10.718065 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 13404
I20260927 23:05:10.733841 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 13535
I20260927 23:05:10.741376 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:10.734647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45868 (local address 127.3.240.194:33955)
0927 23:05:10.735245 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:10.735252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:10.735276 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:05:10.735341 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:10.735349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:10.735557 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:10.735718 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:10.736083 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.737029 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.738283 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.738815 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.738957 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:10.740028 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:10.740036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:10.740048 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:10.740587 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:10.740631 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:10.740648 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:10.740760 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:10.740899 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 23:05:10.741032 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":377,"threads_started":1}
I20260927 23:05:10.742583 13657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:10.822214 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 13535
I20260927 23:05:10.837724 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 13665
I20260927 23:05:10.845703 13805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:10.838692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.3.240.195:42111)
0927 23:05:10.839339 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:10.839346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:10.839374 (+ 28us) server_negotiation.cc:407] Connection header received
0927 23:05:10.839429 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:10.839435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:10.839644 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:10.839829 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:10.840236 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.841250 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.842421 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:10.842968 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:10.843185 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:10.844243 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:10.844252 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:10.844264 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:10.844880 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:10.844923 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:10.844941 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:10.845077 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:10.845231 (+ 154us) server_negotiation.cc:299] Negotiation successful
0927 23:05:10.845360 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":362,"threads_started":1}
I20260927 23:05:10.846985 13787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:10.928124 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 13665
I20260927 23:05:10.944216 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 13314
I20260927 23:05:10.945320 13375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:11.044799 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 13314
2026-09-27T23:05:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2315 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-27T23:05:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:05:11Z Disabled control of system clock
I20260927 23:05:11.113694 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43677
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:43677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 23:05:11.411659 13811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:11.412045 13811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:11.412171 13811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:11.418751 13811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:05:11.418850 13811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:11.418887 13811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:11.418942 13811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:05:11.419003 13811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:05:11.427522 13811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:43677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:11.428627 13811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:11.430341 13811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:11.441120 13816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:11.441238 13819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:11.441256 13817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:11.442191 13811 server_base.cc:1034] running on GCE node
I20260927 23:05:11.442906 13811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:11.444213 13811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:11.445431 13811 hybrid_clock.cc:648] HybridClock initialized: now 1790550311445404 us; error 49 us; skew 500 ppm
I20260927 23:05:11.445742 13811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:05:11.447815 13811 webserver.cc:460] Webserver started at http://127.3.240.254:34647/ using document root <none> and password file <none>
I20260927 23:05:11.448447 13811 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:11.448593 13811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:11.448915 13811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:11.451581 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "839f3f6acc5242ba9704a9e96bdf01c1"
format_stamp: "Formatted at 2026-09-27 23:05:11 on dist-test-slave-lmfp"
server_key: "746d628b2988f2a62d190d317b22c3df"
server_key_iv: "bc06dd86e71c00cdc3e2d0918dea9d3a"
server_key_version: "encryptionkey@0"
I20260927 23:05:11.452330 13811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "839f3f6acc5242ba9704a9e96bdf01c1"
format_stamp: "Formatted at 2026-09-27 23:05:11 on dist-test-slave-lmfp"
server_key: "746d628b2988f2a62d190d317b22c3df"
server_key_iv: "bc06dd86e71c00cdc3e2d0918dea9d3a"
server_key_version: "encryptionkey@0"
I20260927 23:05:11.457057 13811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:05:11.460034 13825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:11.461192 13811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:05:11.461359 13811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "839f3f6acc5242ba9704a9e96bdf01c1"
format_stamp: "Formatted at 2026-09-27 23:05:11 on dist-test-slave-lmfp"
server_key: "746d628b2988f2a62d190d317b22c3df"
server_key_iv: "bc06dd86e71c00cdc3e2d0918dea9d3a"
server_key_version: "encryptionkey@0"
I20260927 23:05:11.461570 13811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:11.466979 13811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:11.467739 13811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:11.468024 13811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:11.484453 13811 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:43677
I20260927 23:05:11.484520 13876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:43677 every 8 connection(s)
I20260927 23:05:11.485553 13811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:05:11.489032 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:05:11.490614 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 13811
I20260927 23:05:11.490825 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:05:11.491483 4035 external_mini_cluster.cc:1428] Setting key 5e4748a103a2d88c0733271b5108e9f5
I20260927 23:05:11.500871 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:11.493420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.3.240.254:43677)
0927 23:05:11.494517 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:11.494531 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:11.494586 (+ 55us) server_negotiation.cc:407] Connection header received
0927 23:05:11.496980 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:11.497004 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:11.497457 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:11.497855 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:11.498880 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:11.498888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:11.498910 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:11.498928 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:11.499025 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:11.499057 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:11.499328 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:11.499464 (+ 136us) server_negotiation.cc:299] Negotiation successful
0927 23:05:11.499733 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":358,"threads_started":1}
I20260927 23:05:11.502218 13877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1: Bootstrap starting.
I20260927 23:05:11.505429 13877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1: Neither blocks nor log segments found. Creating new log.
I20260927 23:05:11.506632 13877 log.cc:824] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1: Log is configured to *not* fsync() on all Append() calls
I20260927 23:05:11.509222 13877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1: No bootstrap required, opened a new log
I20260927 23:05:11.515806 13877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839f3f6acc5242ba9704a9e96bdf01c1" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43677 } }
I20260927 23:05:11.516120 13877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:05:11.516244 13877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 839f3f6acc5242ba9704a9e96bdf01c1, State: Initialized, Role: FOLLOWER
I20260927 23:05:11.516778 13877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839f3f6acc5242ba9704a9e96bdf01c1" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43677 } }
I20260927 23:05:11.517052 13877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:05:11.517204 13877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:05:11.517391 13877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:05:11.521282 13877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839f3f6acc5242ba9704a9e96bdf01c1" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43677 } }
I20260927 23:05:11.521765 13877 leader_election.cc:302] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 839f3f6acc5242ba9704a9e96bdf01c1; no voters:
I20260927 23:05:11.522305 13877 leader_election.cc:288] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:05:11.522439 13882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:05:11.523236 13882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [term 1 LEADER]: Becoming Leader. State: Replica: 839f3f6acc5242ba9704a9e96bdf01c1, State: Running, Role: LEADER
I20260927 23:05:11.523802 13882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839f3f6acc5242ba9704a9e96bdf01c1" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43677 } }
I20260927 23:05:11.524282 13877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:05:11.526357 13884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 839f3f6acc5242ba9704a9e96bdf01c1. Latest consensus state: current_term: 1 leader_uuid: "839f3f6acc5242ba9704a9e96bdf01c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839f3f6acc5242ba9704a9e96bdf01c1" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43677 } } }
I20260927 23:05:11.526328 13883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "839f3f6acc5242ba9704a9e96bdf01c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "839f3f6acc5242ba9704a9e96bdf01c1" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43677 } } }
I20260927 23:05:11.526845 13883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [sys.catalog]: This master's current role is: LEADER
I20260927 23:05:11.526857 13884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1 [sys.catalog]: This master's current role is: LEADER
I20260927 23:05:11.528302 13888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:05:11.532955 13888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:05:11.540630 13888 catalog_manager.cc:1347] Generated new cluster ID: 87ff31044fa5473d9ff09b2a46d2d639
I20260927 23:05:11.540728 13888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:05:11.556102 13888 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:05:11.556469 13888 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:05:11.571425 13888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 839f3f6acc5242ba9704a9e96bdf01c1: Generated new TSK 0
I20260927 23:05:11.572116 13888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:05:11.582762 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43677
--builtin_ntp_servers=127.3.240.212:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260927 23:05:11.895617 13901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:11.895864 13901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:11.895951 13901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:11.902645 13901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:11.902756 13901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:11.902865 13901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:05:11.910912 13901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:11.912058 13901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:11.913666 13901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:11.925997 13906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:11.925961 13909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:11.926079 13907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:11.926935 13901 server_base.cc:1034] running on GCE node
I20260927 23:05:11.927572 13901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:11.928397 13901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:11.929615 13901 hybrid_clock.cc:648] HybridClock initialized: now 1790550311929583 us; error 48 us; skew 500 ppm
I20260927 23:05:11.929922 13901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:05:11.931901 13901 webserver.cc:460] Webserver started at http://127.3.240.193:33593/ using document root <none> and password file <none>
I20260927 23:05:11.932664 13901 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:11.932775 13901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:11.933089 13901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:11.935818 13901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "26ba8de882e84a9fb161655ac18adb16"
format_stamp: "Formatted at 2026-09-27 23:05:11 on dist-test-slave-lmfp"
server_key: "951f8e4f67ab061b2b1f711b8ab7e123"
server_key_iv: "53be99a96d5ece144876f87760d75339"
server_key_version: "encryptionkey@0"
I20260927 23:05:11.936576 13901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "26ba8de882e84a9fb161655ac18adb16"
format_stamp: "Formatted at 2026-09-27 23:05:11 on dist-test-slave-lmfp"
server_key: "951f8e4f67ab061b2b1f711b8ab7e123"
server_key_iv: "53be99a96d5ece144876f87760d75339"
server_key_version: "encryptionkey@0"
I20260927 23:05:11.941108 13901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:05:11.944203 13915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:11.945384 13901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:05:11.945536 13901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "26ba8de882e84a9fb161655ac18adb16"
format_stamp: "Formatted at 2026-09-27 23:05:11 on dist-test-slave-lmfp"
server_key: "951f8e4f67ab061b2b1f711b8ab7e123"
server_key_iv: "53be99a96d5ece144876f87760d75339"
server_key_version: "encryptionkey@0"
I20260927 23:05:11.945732 13901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:11.950982 13901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:11.951790 13901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:11.952044 13901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:11.952854 13901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:11.954195 13901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:11.954306 13901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:11.954417 13901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:11.954495 13901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:11.986038 13901 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:33589
I20260927 23:05:11.986088 14027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:33589 every 8 connection(s)
I20260927 23:05:11.987349 13901 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:05:11.993484 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 13901
I20260927 23:05:11.993685 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:05:11.994446 4035 external_mini_cluster.cc:1428] Setting key bf35a4654d812c3101355b31a09dcb09
I20260927 23:05:11.995280 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:11.989817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:49025 (local address 127.3.240.254:43677)
0927 23:05:11.990018 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:11.990025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:11.991437 (+ 1412us) server_negotiation.cc:407] Connection header received
0927 23:05:11.992956 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:11.992960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:11.993023 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:11.993095 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:11.994543 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:11.994549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:11.994556 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:11.994575 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:11.994609 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:11.994621 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:11.994706 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:11.994926 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 23:05:11.994970 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 23:05:11.996269 14030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:11.990397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43677 (local address 127.3.240.193:49025)
0927 23:05:11.991295 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:11.991341 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:11.992723 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:11.993375 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:11.993388 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:11.993510 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:11.994169 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:05:11.994174 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:11.994226 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:05:11.994345 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:11.994714 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:11.994723 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:11.994731 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 23:05:11.995011 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 23:05:11.995220 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":241,"threads_started":1}
I20260927 23:05:11.997346 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43677
--builtin_ntp_servers=127.3.240.212:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 23:05:11.997572 14028 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43677
I20260927 23:05:11.997854 14028 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:11.998394 14028 heartbeater.cc:511] Master 127.3.240.254:43677 requested a full tablet report, sending...
I20260927 23:05:12.000375 13842 ts_manager.cc:194] Registered new tserver with Master: 26ba8de882e84a9fb161655ac18adb16 (127.3.240.193:33589)
W20260927 23:05:12.295945 14031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:12.296201 14031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:12.296376 14031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:12.302752 14031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:12.302834 14031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:12.303026 14031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:05:12.311209 14031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:12.312335 14031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:12.313990 14031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:12.326043 14037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:12.326265 14036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:12.326296 14039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:12.327159 14031 server_base.cc:1034] running on GCE node
I20260927 23:05:12.327776 14031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:12.328680 14031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:12.329896 14031 hybrid_clock.cc:648] HybridClock initialized: now 1790550312329856 us; error 49 us; skew 500 ppm
I20260927 23:05:12.330199 14031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:05:12.332327 14031 webserver.cc:460] Webserver started at http://127.3.240.194:46047/ using document root <none> and password file <none>
I20260927 23:05:12.332996 14031 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:12.333130 14031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:12.333453 14031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:12.336217 14031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "b2ef72fcf58d48f6892197d983c98d54"
format_stamp: "Formatted at 2026-09-27 23:05:12 on dist-test-slave-lmfp"
server_key: "6dd4cc3fe6361cebc4119079e802bef3"
server_key_iv: "341054f6470194a39446f9de471283c2"
server_key_version: "encryptionkey@0"
I20260927 23:05:12.336989 14031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "b2ef72fcf58d48f6892197d983c98d54"
format_stamp: "Formatted at 2026-09-27 23:05:12 on dist-test-slave-lmfp"
server_key: "6dd4cc3fe6361cebc4119079e802bef3"
server_key_iv: "341054f6470194a39446f9de471283c2"
server_key_version: "encryptionkey@0"
I20260927 23:05:12.341792 14031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:05:12.345091 14045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:12.346311 14031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 23:05:12.346481 14031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "b2ef72fcf58d48f6892197d983c98d54"
format_stamp: "Formatted at 2026-09-27 23:05:12 on dist-test-slave-lmfp"
server_key: "6dd4cc3fe6361cebc4119079e802bef3"
server_key_iv: "341054f6470194a39446f9de471283c2"
server_key_version: "encryptionkey@0"
I20260927 23:05:12.346683 14031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:12.352073 14031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:12.352793 14031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:12.353051 14031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:12.353879 14031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:12.355389 14031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:12.355504 14031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:12.355630 14031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:12.355726 14031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:12.387079 14031 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:46133
I20260927 23:05:12.387169 14157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:46133 every 8 connection(s)
I20260927 23:05:12.388319 14031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:05:12.396096 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:12.390758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:59047 (local address 127.3.240.254:43677)
0927 23:05:12.390933 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:12.390938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:12.392303 (+ 1365us) server_negotiation.cc:407] Connection header received
0927 23:05:12.393919 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:12.393924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:12.393994 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:12.394110 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:12.395352 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:12.395358 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:12.395362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:12.395380 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:12.395418 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:12.395429 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:12.395551 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:12.395741 (+ 190us) server_negotiation.cc:299] Negotiation successful
0927 23:05:12.395805 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:05:12.396776 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 14031
I20260927 23:05:12.396951 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:05:12.397002 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:12.391311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43677 (local address 127.3.240.194:59047)
0927 23:05:12.392146 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:12.392186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:12.393652 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:12.394326 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:12.394334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:12.394427 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:12.395073 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:05:12.395075 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:12.395110 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:05:12.395187 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:12.395524 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:12.395535 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:12.395543 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 23:05:12.395820 (+ 277us) client_negotiation.cc:240] Negotiation successful
0927 23:05:12.396051 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":246,"threads_started":1}
I20260927 23:05:12.397394 4035 external_mini_cluster.cc:1428] Setting key 47fee615cc1c36c1ee3bba53c22894d9
I20260927 23:05:12.398389 14158 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43677
I20260927 23:05:12.398654 14158 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:12.399286 14158 heartbeater.cc:511] Master 127.3.240.254:43677 requested a full tablet report, sending...
I20260927 23:05:12.400207 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43677
--builtin_ntp_servers=127.3.240.212:40347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260927 23:05:12.400638 13842 ts_manager.cc:194] Registered new tserver with Master: b2ef72fcf58d48f6892197d983c98d54 (127.3.240.194:46133)
W20260927 23:05:12.712836 14161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:12.713095 14161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:12.713173 14161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:12.720103 14161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:12.720192 14161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:12.720293 14161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:05:12.729349 14161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:12.730494 14161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:12.732291 14161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:12.744941 14166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:12.745048 14169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:12.745048 14167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:12.745729 14161 server_base.cc:1034] running on GCE node
I20260927 23:05:12.746253 14161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:12.747133 14161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:12.748353 14161 hybrid_clock.cc:648] HybridClock initialized: now 1790550312748307 us; error 49 us; skew 500 ppm
I20260927 23:05:12.748658 14161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:05:12.750866 14161 webserver.cc:460] Webserver started at http://127.3.240.195:40629/ using document root <none> and password file <none>
I20260927 23:05:12.751555 14161 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:12.751672 14161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:12.751992 14161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:12.754531 14161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "7592b594d3844a5fac2161ce770192fa"
format_stamp: "Formatted at 2026-09-27 23:05:12 on dist-test-slave-lmfp"
server_key: "7a991656c5aab617e32782e546853260"
server_key_iv: "97fa0a033506182f913fe7d383f7038d"
server_key_version: "encryptionkey@0"
I20260927 23:05:12.755470 14161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "7592b594d3844a5fac2161ce770192fa"
format_stamp: "Formatted at 2026-09-27 23:05:12 on dist-test-slave-lmfp"
server_key: "7a991656c5aab617e32782e546853260"
server_key_iv: "97fa0a033506182f913fe7d383f7038d"
server_key_version: "encryptionkey@0"
I20260927 23:05:12.760293 14161 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:05:12.763470 14175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:12.764678 14161 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 23:05:12.764832 14161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "7592b594d3844a5fac2161ce770192fa"
format_stamp: "Formatted at 2026-09-27 23:05:12 on dist-test-slave-lmfp"
server_key: "7a991656c5aab617e32782e546853260"
server_key_iv: "97fa0a033506182f913fe7d383f7038d"
server_key_version: "encryptionkey@0"
I20260927 23:05:12.765069 14161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:12.770392 14161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:12.771189 14161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:12.771468 14161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:12.772290 14161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:12.773589 14161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:12.773723 14161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:12.773849 14161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:12.773934 14161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:12.805608 14161 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:35149
I20260927 23:05:12.805689 14287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:35149 every 8 connection(s)
I20260927 23:05:12.806871 14161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:05:12.810210 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 14161
I20260927 23:05:12.810400 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:05:12.810928 4035 external_mini_cluster.cc:1428] Setting key 50b33c7cef809c3dc90da8cf6caf184a
I20260927 23:05:12.815109 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:12.809407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:58763 (local address 127.3.240.254:43677)
0927 23:05:12.809605 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:12.809610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:12.811116 (+ 1506us) server_negotiation.cc:407] Connection header received
0927 23:05:12.812961 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:12.812966 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:12.813029 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:12.813145 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:12.814355 (+ 1210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:12.814361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:12.814365 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:12.814383 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:12.814421 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:12.814434 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:12.814560 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:12.814756 (+ 196us) server_negotiation.cc:299] Negotiation successful
0927 23:05:12.814817 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:05:12.815964 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:12.809813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43677 (local address 127.3.240.195:58763)
0927 23:05:12.810914 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:12.810963 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:12.812710 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:12.813383 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:12.813392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:12.813479 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:12.814088 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:05:12.814090 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:12.814126 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:05:12.814203 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:12.814540 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:12.814551 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:12.814559 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 23:05:12.814834 (+ 275us) client_negotiation.cc:240] Negotiation successful
0927 23:05:12.815097 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":308,"threads_started":1}
I20260927 23:05:12.817060 14288 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43677
I20260927 23:05:12.817286 14288 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:12.817796 14288 heartbeater.cc:511] Master 127.3.240.254:43677 requested a full tablet report, sending...
I20260927 23:05:12.818977 13842 ts_manager.cc:194] Registered new tserver with Master: 7592b594d3844a5fac2161ce770192fa (127.3.240.195:35149)
I20260927 23:05:12.825691 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260927 23:05:12.831125 13880 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:05:12.829617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.3.240.254:43677)
0927 23:05:12.829776 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:12.829782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:12.830268 (+ 486us) server_negotiation.cc:407] Connection header received
0927 23:05:12.830402 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:12.830406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:12.830447 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:12.830518 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:12.830837 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60984: BlockingRecv error: recv got EOF from 127.0.0.1:60984 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 23:05:12.831898 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 13901
I20260927 23:05:12.835134 14297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:12.832799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.3.240.193:33589)
0927 23:05:12.833300 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:12.833305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:12.833327 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:05:12.833380 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:12.833385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:12.833556 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:12.833662 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:12.834449 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:12.834454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:12.834463 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:12.834481 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:12.834515 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:12.834526 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:12.834621 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:12.834693 (+ 72us) server_negotiation.cc:299] Negotiation successful
0927 23:05:12.834786 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":242,"threads_started":1}
I20260927 23:05:12.836185 14023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:12.914773 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 13901
I20260927 23:05:12.930727 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 14031
I20260927 23:05:12.934221 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:12.931636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.3.240.194:46133)
0927 23:05:12.932140 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:12.932146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:12.932170 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:05:12.932270 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:12.932277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:12.932503 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:12.932620 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:12.933526 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:12.933533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:12.933544 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:12.933562 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:12.933600 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:12.933632 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:12.933733 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:12.933815 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 23:05:12.933936 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":240,"threads_started":1}
I20260927 23:05:12.935439 14153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:13.013378 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 14031
I20260927 23:05:13.029882 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 14161
I20260927 23:05:13.033566 14301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:13.030790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.3.240.195:35149)
0927 23:05:13.031425 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:13.031432 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:13.031462 (+ 30us) server_negotiation.cc:407] Connection header received
0927 23:05:13.031518 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:13.031524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:13.031733 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:13.031868 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:13.032993 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:13.032999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:13.033010 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:05:13.033025 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:13.033060 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:05:13.033073 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:13.033194 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:13.033245 (+ 51us) server_negotiation.cc:299] Negotiation successful
0927 23:05:13.033317 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":357,"threads_started":1}
I20260927 23:05:13.034729 14283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:13.116751 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 14161
I20260927 23:05:13.133128 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 13811
I20260927 23:05:13.134310 13872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:13.231251 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 13811
2026-09-27T23:05:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2187 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:05:13 dist-test-slave-lmfp krb5kdc[14306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:05:13 dist-test-slave-lmfp krb5kdc[14306](info): set up 2 sockets
Sep 27 23:05:13 dist-test-slave-lmfp krb5kdc[14306](info): commencing operation
krb5kdc: starting...
W20260927 23:05:14.284119 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:05:14 dist-test-slave-lmfp krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:05:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:05:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:05:14.484354 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40559
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:40559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:05:14.781796 14319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:14.782039 14319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:14.782120 14319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:14.788941 14319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:05:14.789032 14319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:14.789076 14319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:14.789132 14319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:05:14.789175 14319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:05:14.798197 14319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:40559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:14.799436 14319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:14.800684 14319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:14.812474 14325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:14.812520 14327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:14.812546 14324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:14.813159 14319 server_base.cc:1034] running on GCE node
I20260927 23:05:14.813850 14319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:14.815215 14319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:14.816421 14319 hybrid_clock.cc:648] HybridClock initialized: now 1790550314816387 us; error 46 us; skew 500 ppm
I20260927 23:05:14.816717 14319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:05:14 dist-test-slave-lmfp krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550314, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:05:14.825155 14319 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:05:14.826498 14319 webserver.cc:460] Webserver started at http://127.3.240.254:34925/ using document root <none> and password file <none>
I20260927 23:05:14.827250 14319 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:14.827404 14319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:14.827687 14319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:14.830440 14319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "2dd00b263eba4b54b3f4786843c0ff21"
format_stamp: "Formatted at 2026-09-27 23:05:14 on dist-test-slave-lmfp"
server_key: "d6e0972acda98e89e68bca1531b83e6e"
server_key_iv: "5d09805be892f1e26dac5f3e0a44d4f4"
server_key_version: "encryptionkey@0"
I20260927 23:05:14.831270 14319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2dd00b263eba4b54b3f4786843c0ff21"
format_stamp: "Formatted at 2026-09-27 23:05:14 on dist-test-slave-lmfp"
server_key: "d6e0972acda98e89e68bca1531b83e6e"
server_key_iv: "5d09805be892f1e26dac5f3e0a44d4f4"
server_key_version: "encryptionkey@0"
I20260927 23:05:14.836210 14319 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 23:05:14.839977 14334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:14.841203 14319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 23:05:14.841372 14319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "2dd00b263eba4b54b3f4786843c0ff21"
format_stamp: "Formatted at 2026-09-27 23:05:14 on dist-test-slave-lmfp"
server_key: "d6e0972acda98e89e68bca1531b83e6e"
server_key_iv: "5d09805be892f1e26dac5f3e0a44d4f4"
server_key_version: "encryptionkey@0"
I20260927 23:05:14.841557 14319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:14.866483 14319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:14.870478 14319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:14.870728 14319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:14.889015 14319 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:40559
I20260927 23:05:14.889055 14385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:40559 every 8 connection(s)
I20260927 23:05:14.890231 14319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:05:14.891436 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 14319
I20260927 23:05:14.891667 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:05:14.892442 4035 external_mini_cluster.cc:1428] Setting key fccabd00e783a4a3cca1e03f1b921444
I20260927 23:05:14.894760 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 23:05:14 dist-test-slave-lmfp krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:05:14.907581 14386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21: Bootstrap starting.
I20260927 23:05:14.911607 14386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21: Neither blocks nor log segments found. Creating new log.
I20260927 23:05:14.912873 14386 log.cc:824] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21: Log is configured to *not* fsync() on all Append() calls
I20260927 23:05:14.915194 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:14.894416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.3.240.254:40559)
0927 23:05:14.895232 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:14.895244 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:14.895282 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:05:14.896798 (+ 1516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:14.896818 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:14.897173 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:14.897490 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:14.899239 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:14.900556 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:14.902218 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:14.902817 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:14.902937 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:14.907459 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:14.907469 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:14.907485 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:14.907524 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:14.911285 (+ 3761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:14.911959 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:14.911969 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:14.911979 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:14.912179 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:14.912518 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:14.912523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:14.912526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:14.913156 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:14.913549 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:14.913791 (+ 242us) server_negotiation.cc:299] Negotiation successful
0927 23:05:14.914070 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":397,"threads_started":1}
I20260927 23:05:14.915917 14386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21: No bootstrap required, opened a new log
I20260927 23:05:14.923204 14386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd00b263eba4b54b3f4786843c0ff21" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40559 } }
I20260927 23:05:14.923498 14386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:05:14.923568 14386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dd00b263eba4b54b3f4786843c0ff21, State: Initialized, Role: FOLLOWER
I20260927 23:05:14.924093 14386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd00b263eba4b54b3f4786843c0ff21" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40559 } }
I20260927 23:05:14.924276 14386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:05:14.924369 14386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:05:14.924476 14386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:05:14.926385 14386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd00b263eba4b54b3f4786843c0ff21" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40559 } }
I20260927 23:05:14.926813 14386 leader_election.cc:302] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dd00b263eba4b54b3f4786843c0ff21; no voters:
I20260927 23:05:14.927340 14386 leader_election.cc:288] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:05:14.927466 14391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:05:14.928650 14391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [term 1 LEADER]: Becoming Leader. State: Replica: 2dd00b263eba4b54b3f4786843c0ff21, State: Running, Role: LEADER
I20260927 23:05:14.929118 14386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:05:14.929118 14391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd00b263eba4b54b3f4786843c0ff21" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40559 } }
I20260927 23:05:14.931846 14393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dd00b263eba4b54b3f4786843c0ff21. Latest consensus state: current_term: 1 leader_uuid: "2dd00b263eba4b54b3f4786843c0ff21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd00b263eba4b54b3f4786843c0ff21" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40559 } } }
I20260927 23:05:14.931866 14392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dd00b263eba4b54b3f4786843c0ff21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dd00b263eba4b54b3f4786843c0ff21" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40559 } } }
I20260927 23:05:14.932220 14392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [sys.catalog]: This master's current role is: LEADER
I20260927 23:05:14.932220 14393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21 [sys.catalog]: This master's current role is: LEADER
I20260927 23:05:14.933041 14399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:05:14.938290 14399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:05:14.946264 14399 catalog_manager.cc:1347] Generated new cluster ID: c99a53a6305a48f5a8962e6121af10a3
I20260927 23:05:14.946375 14399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:05:14.975097 14399 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:05:14.976923 14399 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:05:14.984577 14399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21: Generated new TSK 0
I20260927 23:05:14.985483 14399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:05:15.034541 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40559
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:05:15.346964 14412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:15.347295 14412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:15.347465 14412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:15.354828 14412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:15.354947 14412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:15.355136 14412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:05:15.364885 14412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:15.366034 14412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:15.367326 14412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:15.379457 14420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:15.379529 14418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:15.379541 14417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:15.380257 14412 server_base.cc:1034] running on GCE node
I20260927 23:05:15.380749 14412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:15.381558 14412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:15.382758 14412 hybrid_clock.cc:648] HybridClock initialized: now 1790550315382725 us; error 22 us; skew 500 ppm
I20260927 23:05:15.383105 14412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:05:15 dist-test-slave-lmfp krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:05:15.385915 14412 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:05:15.387181 14412 webserver.cc:460] Webserver started at http://127.3.240.193:41063/ using document root <none> and password file <none>
I20260927 23:05:15.387766 14412 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:15.387858 14412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:15.388079 14412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:15.390631 14412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "2e66f6e0e88547b395c5d2165f3c20ff"
format_stamp: "Formatted at 2026-09-27 23:05:15 on dist-test-slave-lmfp"
server_key: "85bb6ea4b7304024cf0224f6ade08931"
server_key_iv: "e582bbffded00f3c1c5ea230fa9c0774"
server_key_version: "encryptionkey@0"
I20260927 23:05:15.391434 14412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "2e66f6e0e88547b395c5d2165f3c20ff"
format_stamp: "Formatted at 2026-09-27 23:05:15 on dist-test-slave-lmfp"
server_key: "85bb6ea4b7304024cf0224f6ade08931"
server_key_iv: "e582bbffded00f3c1c5ea230fa9c0774"
server_key_version: "encryptionkey@0"
I20260927 23:05:15.396065 14412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 23:05:15.399290 14427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:15.400467 14412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:05:15.400624 14412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "2e66f6e0e88547b395c5d2165f3c20ff"
format_stamp: "Formatted at 2026-09-27 23:05:15 on dist-test-slave-lmfp"
server_key: "85bb6ea4b7304024cf0224f6ade08931"
server_key_iv: "e582bbffded00f3c1c5ea230fa9c0774"
server_key_version: "encryptionkey@0"
I20260927 23:05:15.400796 14412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:15.419544 14412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:15.423152 14412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:15.423421 14412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:15.424263 14412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:15.425453 14412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:15.425518 14412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:15.425679 14412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:15.425736 14412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:15.457297 14412 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:36885
I20260927 23:05:15.457343 14539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:36885 every 8 connection(s)
I20260927 23:05:15.458473 14412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:05:15.464500 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 14412
I20260927 23:05:15.464726 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:05:15.465517 4035 external_mini_cluster.cc:1428] Setting key af91448e9d1a6a0ee5280edc87caa31b
Sep 27 23:05:15 dist-test-slave-lmfp krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:05:15.479879 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:15.460810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:36269 (local address 127.3.240.254:40559)
0927 23:05:15.461239 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:15.461245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:15.462194 (+ 949us) server_negotiation.cc:407] Connection header received
0927 23:05:15.463929 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:15.463933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:15.463993 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:15.464083 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:15.466050 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:15.466890 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:15.469271 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:15.469638 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:15.469725 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:15.473960 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:15.473969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:15.473977 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:15.474051 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:15.477009 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:15.477864 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:15.477867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:15.477869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:15.477969 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:15.478390 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:15.478392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:15.478394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:15.478616 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:15.478711 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:15.479504 (+ 793us) server_negotiation.cc:299] Negotiation successful
0927 23:05:15.479646 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
I20260927 23:05:15.481328 14543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:15.461363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40559 (local address 127.3.240.193:36269)
0927 23:05:15.462037 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:15.462080 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:15.463687 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:15.464389 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:15.464401 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:15.465112 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:15.465892 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:15.465916 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:15.467098 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:15.467119 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:15.469118 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:15.469136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:15.469852 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:15.469861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:15.470168 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:15.470849 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:15.470884 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:15.473729 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:15.477140 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:05:15.477152 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:05:15.477171 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:05:15.477756 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:05:15.478062 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:05:15.478068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:05:15.478074 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:05:15.478288 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:05:15.478728 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:15.478738 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:15.479161 (+ 423us) client_negotiation.cc:769] Sending connection context
0927 23:05:15.479439 (+ 278us) client_negotiation.cc:240] Negotiation successful
0927 23:05:15.479959 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":245,"threads_started":1}
I20260927 23:05:15.482784 14540 heartbeater.cc:347] Connected to a master server at 127.3.240.254:40559
I20260927 23:05:15.483119 14540 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:15.483804 14540 heartbeater.cc:511] Master 127.3.240.254:40559 requested a full tablet report, sending...
I20260927 23:05:15.486061 14351 ts_manager.cc:194] Registered new tserver with Master: 2e66f6e0e88547b395c5d2165f3c20ff (127.3.240.193:36885)
I20260927 23:05:15.488201 14351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:36269
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:05:15.513681 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40559
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:05:15.817185 14546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:15.817442 14546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:15.817593 14546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:15.824676 14546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:15.824770 14546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:15.824950 14546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:05:15.834290 14546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:15.835467 14546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:15.836621 14546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:15.848815 14552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:15.848860 14551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:15.848884 14554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:15.849436 14546 server_base.cc:1034] running on GCE node
I20260927 23:05:15.850005 14546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:15.850898 14546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:15.852165 14546 hybrid_clock.cc:648] HybridClock initialized: now 1790550315852129 us; error 31 us; skew 500 ppm
I20260927 23:05:15.852483 14546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:05:15 dist-test-slave-lmfp krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:05:15.855355 14546 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:05:15.856581 14546 webserver.cc:460] Webserver started at http://127.3.240.194:36643/ using document root <none> and password file <none>
I20260927 23:05:15.857223 14546 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:15.857367 14546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:15.857667 14546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:15.860508 14546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "f4a515794a0c49b9a53d5658d5a2ffd1"
format_stamp: "Formatted at 2026-09-27 23:05:15 on dist-test-slave-lmfp"
server_key: "9596fb275f57226ccbdd2be39a54eb70"
server_key_iv: "a0472b136fd15bb34d11fc043ff12d33"
server_key_version: "encryptionkey@0"
I20260927 23:05:15.861228 14546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "f4a515794a0c49b9a53d5658d5a2ffd1"
format_stamp: "Formatted at 2026-09-27 23:05:15 on dist-test-slave-lmfp"
server_key: "9596fb275f57226ccbdd2be39a54eb70"
server_key_iv: "a0472b136fd15bb34d11fc043ff12d33"
server_key_version: "encryptionkey@0"
I20260927 23:05:15.866171 14546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 23:05:15.869460 14561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:15.870663 14546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 23:05:15.870821 14546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "f4a515794a0c49b9a53d5658d5a2ffd1"
format_stamp: "Formatted at 2026-09-27 23:05:15 on dist-test-slave-lmfp"
server_key: "9596fb275f57226ccbdd2be39a54eb70"
server_key_iv: "a0472b136fd15bb34d11fc043ff12d33"
server_key_version: "encryptionkey@0"
I20260927 23:05:15.871073 14546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:15.892211 14546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:15.896405 14546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:15.896677 14546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:15.897596 14546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:15.898945 14546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:15.899060 14546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 23:05:15.899197 14546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:15.899282 14546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:15.930836 14546 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:43259
I20260927 23:05:15.930915 14673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:43259 every 8 connection(s)
I20260927 23:05:15.932082 14546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:05:15.942337 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 14546
I20260927 23:05:15.942528 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:05:15.942934 4035 external_mini_cluster.cc:1428] Setting key bfbcd10d757d0846e1f701c9b07ec15a
Sep 27 23:05:15 dist-test-slave-lmfp krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:05:15.951268 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:15.934518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:58051 (local address 127.3.240.254:40559)
0927 23:05:15.934727 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:15.934734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:15.935892 (+ 1158us) server_negotiation.cc:407] Connection header received
0927 23:05:15.937512 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:15.937516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:15.937577 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:15.937661 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:15.938970 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:15.939750 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:15.941105 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:15.941446 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:15.941527 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:15.945104 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:15.945109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:15.945114 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:15.945146 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:15.948490 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:15.949138 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:15.949146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:15.949152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:15.949301 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:15.949656 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:15.949663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:15.949668 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:15.950000 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:15.950136 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:15.950694 (+ 558us) server_negotiation.cc:299] Negotiation successful
0927 23:05:15.950911 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 23:05:15.952210 14676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:15.935104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40559 (local address 127.3.240.194:58051)
0927 23:05:15.935717 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:15.935755 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:15.937285 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:15.937889 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:15.937897 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:15.938333 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:15.938834 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:15.938849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:15.939873 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:15.939876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:15.940999 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:15.941009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:15.941611 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:15.941615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:15.941776 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:15.942472 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:15.942517 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:15.944893 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:15.948607 (+ 3714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:05:15.948612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:05:15.948623 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:05:15.949027 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:05:15.949400 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:05:15.949403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:05:15.949405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:05:15.949558 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:05:15.950112 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:15.950117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:15.950375 (+ 258us) client_negotiation.cc:769] Sending connection context
0927 23:05:15.950622 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 23:05:15.951094 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":237,"threads_started":1}
I20260927 23:05:15.953472 14674 heartbeater.cc:347] Connected to a master server at 127.3.240.254:40559
I20260927 23:05:15.953743 14674 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:15.954483 14674 heartbeater.cc:511] Master 127.3.240.254:40559 requested a full tablet report, sending...
I20260927 23:05:15.955888 14351 ts_manager.cc:194] Registered new tserver with Master: f4a515794a0c49b9a53d5658d5a2ffd1 (127.3.240.194:43259)
I20260927 23:05:15.956614 14351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:58051
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:05:15.991243 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40559
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:05:16.296591 14679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:16.296888 14679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:16.297052 14679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:16.303968 14679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:16.304096 14679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:16.304273 14679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:05:16.313493 14679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:16.314738 14679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:16.316025 14679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:16.328076 14684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:16.328183 14685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:16.328461 14687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:16.328927 14679 server_base.cc:1034] running on GCE node
I20260927 23:05:16.329540 14679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:16.330498 14679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:16.331713 14679 hybrid_clock.cc:648] HybridClock initialized: now 1790550316331671 us; error 34 us; skew 500 ppm
I20260927 23:05:16.332020 14679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:05:16 dist-test-slave-lmfp krb5kdc[14306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:05:16.334918 14679 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:05:16.336165 14679 webserver.cc:460] Webserver started at http://127.3.240.195:36837/ using document root <none> and password file <none>
I20260927 23:05:16.336786 14679 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:16.336920 14679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:16.337234 14679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:05:16.340049 14679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "2a921a4ed67849e28a9f2acf723d4fc9"
format_stamp: "Formatted at 2026-09-27 23:05:16 on dist-test-slave-lmfp"
server_key: "4411c6b3d537c27832c8ba4fe5308de1"
server_key_iv: "bcceddef3f14a58a850016b65c61c8a2"
server_key_version: "encryptionkey@0"
I20260927 23:05:16.340816 14679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "2a921a4ed67849e28a9f2acf723d4fc9"
format_stamp: "Formatted at 2026-09-27 23:05:16 on dist-test-slave-lmfp"
server_key: "4411c6b3d537c27832c8ba4fe5308de1"
server_key_iv: "bcceddef3f14a58a850016b65c61c8a2"
server_key_version: "encryptionkey@0"
I20260927 23:05:16.345719 14679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 23:05:16.349403 14694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:16.350659 14679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:05:16.350859 14679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "2a921a4ed67849e28a9f2acf723d4fc9"
format_stamp: "Formatted at 2026-09-27 23:05:16 on dist-test-slave-lmfp"
server_key: "4411c6b3d537c27832c8ba4fe5308de1"
server_key_iv: "bcceddef3f14a58a850016b65c61c8a2"
server_key_version: "encryptionkey@0"
I20260927 23:05:16.351080 14679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:16.364570 14679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:16.369164 14679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:16.369501 14679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:16.371802 14679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:05:16.374600 14679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:05:16.374750 14679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 23:05:16.374918 14679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:05:16.375025 14679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:16.409854 14679 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33933
I20260927 23:05:16.409873 14806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33933 every 8 connection(s)
I20260927 23:05:16.411116 14679 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:05:16.414577 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 14679
I20260927 23:05:16.414764 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:05:16.415383 4035 external_mini_cluster.cc:1428] Setting key 6e3bec99ff1de85218e29065cf1aa7cb
Sep 27 23:05:16 dist-test-slave-lmfp krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550316, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:05:16.432919 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.413786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38091 (local address 127.3.240.254:40559)
0927 23:05:16.413954 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.413958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:16.415541 (+ 1583us) server_negotiation.cc:407] Connection header received
0927 23:05:16.417826 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:16.417830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:16.417908 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:16.418018 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:16.419592 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.420304 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.421749 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.422149 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.422282 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.425982 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:16.425987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:16.425990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:16.426020 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:16.429159 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:16.430266 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:16.430272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:16.430275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:16.430460 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:16.431064 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:16.431069 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:16.431073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:16.431378 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:16.431499 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:16.432483 (+ 984us) server_negotiation.cc:299] Negotiation successful
0927 23:05:16.432645 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 23:05:16.434636 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.414488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40559 (local address 127.3.240.195:38091)
0927 23:05:16.415362 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:16.415415 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:16.417569 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:16.418241 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:16.418253 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:16.418853 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:05:16.419428 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.419444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.420450 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.420455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.421607 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.421625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.422362 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.422370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.422612 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.423378 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:05:16.423400 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:05:16.425756 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:05:16.429326 (+ 3570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:05:16.429338 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:05:16.429362 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:05:16.430118 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:05:16.430584 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:05:16.430592 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:05:16.430598 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:05:16.430859 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:05:16.431519 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:05:16.431530 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:05:16.432040 (+ 510us) client_negotiation.cc:769] Sending connection context
0927 23:05:16.432399 (+ 359us) client_negotiation.cc:240] Negotiation successful
0927 23:05:16.433040 (+ 641us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":330,"threads_started":1}
I20260927 23:05:16.436174 14807 heartbeater.cc:347] Connected to a master server at 127.3.240.254:40559
I20260927 23:05:16.436482 14807 heartbeater.cc:464] Registering TS with master...
I20260927 23:05:16.437170 14807 heartbeater.cc:511] Master 127.3.240.254:40559 requested a full tablet report, sending...
I20260927 23:05:16.438516 14351 ts_manager.cc:194] Registered new tserver with Master: 2a921a4ed67849e28a9f2acf723d4fc9 (127.3.240.195:33933)
I20260927 23:05:16.439360 14351 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:38091
I20260927 23:05:16.442963 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:05:16.461351 14542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.447717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.3.240.254:40559)
0927 23:05:16.447933 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.447941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:16.448442 (+ 501us) server_negotiation.cc:407] Connection header received
0927 23:05:16.448599 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:16.448602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:16.448651 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:16.448717 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:16.450645 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.451428 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.453654 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.454065 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.454177 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.456638 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:16.456642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:16.456646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:16.456673 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:16.459031 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:16.459573 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:16.459579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:16.459582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:16.459696 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:16.460057 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:16.460061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:16.460064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:16.460365 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:16.460582 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:16.460875 (+ 293us) server_negotiation.cc:299] Negotiation successful
0927 23:05:16.461081 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:05:16.465274 14351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:05:16.471812 14351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:05:16.491180 14540 heartbeater.cc:503] Master 127.3.240.254:40559 was elected leader, sending a full tablet report...
I20260927 23:05:16.515911 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.484975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.3.240.194:43259)
0927 23:05:16.505065 (+ 20090us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.505077 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:16.505106 (+ 29us) server_negotiation.cc:407] Connection header received
0927 23:05:16.505433 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:16.505443 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:16.505709 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:16.505893 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:05:16.506891 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.508731 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.512030 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.513793 (+ 1763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.514023 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.514289 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:16.515257 (+ 968us) server_negotiation.cc:299] Negotiation successful
0927 23:05:16.515467 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19903,"thread_start_us":20236,"threads_started":1}
I20260927 23:05:16.517190 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.484644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:43259 (local address 127.0.0.1:36738)
0927 23:05:16.499621 (+ 14977us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:16.499664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:16.499861 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:16.506214 (+ 6353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:16.506223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:16.506275 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:05:16.506673 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.506686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.508921 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.508928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.511831 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.511853 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.514503 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.514515 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.514785 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.514965 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 23:05:16.516462 (+ 1497us) client_negotiation.cc:240] Negotiation successful
0927 23:05:16.516706 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14700,"mutex_wait_us":88,"thread_start_us":15406,"threads_started":1}
I20260927 23:05:16.518708 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.484998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33933 (local address 127.0.0.1:48156)
0927 23:05:16.506323 (+ 21325us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:16.506339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:16.506443 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:16.509205 (+ 2762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:16.509209 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:16.509231 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:05:16.509778 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.509794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.512923 (+ 3129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.512928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.515086 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.515100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.517926 (+ 2826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.517932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.518094 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.518127 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 23:05:16.518193 (+ 66us) client_negotiation.cc:240] Negotiation successful
0927 23:05:16.518327 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21196,"mutex_wait_us":3,"thread_start_us":26768,"threads_started":1}
I20260927 23:05:16.519451 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.496218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36885 (local address 127.0.0.1:40194)
0927 23:05:16.510759 (+ 14541us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:16.510775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:16.510922 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:16.511974 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:16.511979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:16.512001 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:05:16.512330 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.512340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.514367 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.514371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.516542 (+ 2171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.516555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.518622 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.518628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.518942 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.518970 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 23:05:16.519065 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 23:05:16.519182 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14405,"thread_start_us":9857,"threads_started":1}
I20260927 23:05:16.519815 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.485582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.3.240.195:33933)
0927 23:05:16.508767 (+ 23185us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.508773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:16.508796 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:05:16.508867 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:16.508871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:16.509026 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:16.509213 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:05:16.510044 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.512772 (+ 2728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.515290 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.517719 (+ 2429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.518922 (+ 1203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.519157 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:16.519217 (+ 60us) server_negotiation.cc:299] Negotiation successful
0927 23:05:16.519381 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23054,"thread_start_us":22913,"threads_started":1}
I20260927 23:05:16.520112 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.496844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40194 (local address 127.3.240.193:36885)
0927 23:05:16.506958 (+ 10114us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.506969 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:16.511051 (+ 4082us) server_negotiation.cc:407] Connection header received
0927 23:05:16.511182 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:16.511189 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:16.511410 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:16.511589 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:05:16.512493 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.514208 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.516704 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.518157 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.518283 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.518491 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:16.519671 (+ 1180us) server_negotiation.cc:299] Negotiation successful
0927 23:05:16.519830 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9954,"thread_start_us":9836,"threads_started":1}
I20260927 23:05:16.521864 14609 tablet_service.cc:1467] Processing CreateTablet for tablet 6419d711102f4f40ba24d82f074909b7 (DEFAULT_TABLE table=test-table [id=42410ab4607f4d8bb64937db50e90484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:05:16.523496 14609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6419d711102f4f40ba24d82f074909b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:05:16.523293 14742 tablet_service.cc:1467] Processing CreateTablet for tablet 6419d711102f4f40ba24d82f074909b7 (DEFAULT_TABLE table=test-table [id=42410ab4607f4d8bb64937db50e90484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:05:16.523298 14475 tablet_service.cc:1467] Processing CreateTablet for tablet 6419d711102f4f40ba24d82f074909b7 (DEFAULT_TABLE table=test-table [id=42410ab4607f4d8bb64937db50e90484]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:05:16.524586 14742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6419d711102f4f40ba24d82f074909b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:05:16.524591 14475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6419d711102f4f40ba24d82f074909b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:05:16.548666 14826 tablet_bootstrap.cc:492] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Bootstrap starting.
I20260927 23:05:16.553136 14826 tablet_bootstrap.cc:654] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Neither blocks nor log segments found. Creating new log.
I20260927 23:05:16.546025 14825 tablet_bootstrap.cc:492] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Bootstrap starting.
I20260927 23:05:16.554147 14827 tablet_bootstrap.cc:492] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Bootstrap starting.
I20260927 23:05:16.554379 14826 log.cc:824] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Log is configured to *not* fsync() on all Append() calls
I20260927 23:05:16.557381 14826 tablet_bootstrap.cc:492] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: No bootstrap required, opened a new log
I20260927 23:05:16.557647 14826 ts_tablet_manager.cc:1397] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 23:05:16.558230 14827 tablet_bootstrap.cc:654] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Neither blocks nor log segments found. Creating new log.
I20260927 23:05:16.558580 14825 tablet_bootstrap.cc:654] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Neither blocks nor log segments found. Creating new log.
I20260927 23:05:16.559398 14827 log.cc:824] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Log is configured to *not* fsync() on all Append() calls
I20260927 23:05:16.559928 14825 log.cc:824] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Log is configured to *not* fsync() on all Append() calls
I20260927 23:05:16.566452 14825 tablet_bootstrap.cc:492] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: No bootstrap required, opened a new log
I20260927 23:05:16.566731 14825 ts_tablet_manager.cc:1397] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20260927 23:05:16.568487 14827 tablet_bootstrap.cc:492] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: No bootstrap required, opened a new log
I20260927 23:05:16.568765 14827 ts_tablet_manager.cc:1397] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.004s
I20260927 23:05:16.568712 14826 raft_consensus.cc:348] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.569115 14826 raft_consensus.cc:374] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:05:16.569221 14826 raft_consensus.cc:729] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a515794a0c49b9a53d5658d5a2ffd1, State: Initialized, Role: FOLLOWER
I20260927 23:05:16.569818 14826 consensus_queue.cc:260] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.579965 14825 raft_consensus.cc:348] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.580402 14825 raft_consensus.cc:374] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:05:16.580510 14825 raft_consensus.cc:729] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e66f6e0e88547b395c5d2165f3c20ff, State: Initialized, Role: FOLLOWER
I20260927 23:05:16.581128 14674 heartbeater.cc:503] Master 127.3.240.254:40559 was elected leader, sending a full tablet report...
I20260927 23:05:16.581305 14825 consensus_queue.cc:260] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.581764 14826 ts_tablet_manager.cc:1428] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Time spent starting tablet: real 0.024s user 0.013s sys 0.010s
I20260927 23:05:16.583446 14827 raft_consensus.cc:348] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.583875 14827 raft_consensus.cc:374] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:05:16.584016 14827 raft_consensus.cc:729] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a921a4ed67849e28a9f2acf723d4fc9, State: Initialized, Role: FOLLOWER
I20260927 23:05:16.584589 14827 consensus_queue.cc:260] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.588510 14807 heartbeater.cc:503] Master 127.3.240.254:40559 was elected leader, sending a full tablet report...
I20260927 23:05:16.589154 14827 ts_tablet_manager.cc:1428] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Time spent starting tablet: real 0.020s user 0.011s sys 0.006s
I20260927 23:05:16.589339 14825 ts_tablet_manager.cc:1428] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Time spent starting tablet: real 0.022s user 0.018s sys 0.003s
W20260927 23:05:16.662810 14808 tablet.cc:2369] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:05:16.684087 14675 tablet.cc:2369] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:05:16.710703 14541 tablet.cc:2369] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:05:16.845839 14831 raft_consensus.cc:482] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:05:16.846159 14831 raft_consensus.cc:504] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.847847 14831 leader_election.cc:288] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a921a4ed67849e28a9f2acf723d4fc9 (127.3.240.195:33933), 2e66f6e0e88547b395c5d2165f3c20ff (127.3.240.193:36885)
I20260927 23:05:16.855296 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.848303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:58255 (local address 127.3.240.195:33933)
0927 23:05:16.848657 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.848665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:16.849158 (+ 493us) server_negotiation.cc:407] Connection header received
0927 23:05:16.849372 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:16.849376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:16.849429 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:16.849523 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:05:16.850073 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.850749 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.853529 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.854348 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.854475 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.854524 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:16.854847 (+ 323us) server_negotiation.cc:299] Negotiation successful
0927 23:05:16.854988 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
I20260927 23:05:16.855474 14835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.848118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33933 (local address 127.3.240.194:58255)
0927 23:05:16.849054 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:16.849068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:16.849234 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:16.849537 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:16.849541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:16.849562 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:05:16.849938 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.849948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.850870 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.850877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.853342 (+ 2465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.853355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.854541 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.854547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.854701 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.854725 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 23:05:16.854792 (+ 67us) client_negotiation.cc:240] Negotiation successful
0927 23:05:16.854883 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":114,"spinlock_wait_cycles":31104,"thread_start_us":493,"threads_started":1}
I20260927 23:05:16.855504 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.848266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:53311 (local address 127.3.240.193:36885)
0927 23:05:16.848503 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.848511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:16.849276 (+ 765us) server_negotiation.cc:407] Connection header received
0927 23:05:16.849495 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:16.849498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:16.849559 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:16.849672 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:05:16.850315 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.850980 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.853539 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.854427 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.854580 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.854645 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:16.855138 (+ 493us) server_negotiation.cc:299] Negotiation successful
0927 23:05:16.855259 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 23:05:16.855880 14834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.848253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36885 (local address 127.3.240.194:53311)
0927 23:05:16.849146 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0927 23:05:16.849161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:05:16.849342 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:05:16.849685 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:05:16.849690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:05:16.849742 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:05:16.850158 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.850167 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.851161 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.851167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.853342 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:05:16.853355 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.854722 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.854733 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:05:16.854927 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.854978 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 23:05:16.855110 (+ 132us) client_negotiation.cc:240] Negotiation successful
0927 23:05:16.855224 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":83,"spinlock_wait_cycles":896,"thread_start_us":476,"threads_started":1}
I20260927 23:05:16.856664 14762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6419d711102f4f40ba24d82f074909b7" candidate_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" is_pre_election: true
I20260927 23:05:16.856830 14495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6419d711102f4f40ba24d82f074909b7" candidate_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" is_pre_election: true
I20260927 23:05:16.857167 14762 raft_consensus.cc:2454] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a515794a0c49b9a53d5658d5a2ffd1 in term 0.
I20260927 23:05:16.857259 14495 raft_consensus.cc:2454] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a515794a0c49b9a53d5658d5a2ffd1 in term 0.
I20260927 23:05:16.857877 14563 leader_election.cc:302] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a921a4ed67849e28a9f2acf723d4fc9, 2e66f6e0e88547b395c5d2165f3c20ff, f4a515794a0c49b9a53d5658d5a2ffd1; no voters:
I20260927 23:05:16.858268 14831 raft_consensus.cc:2781] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:05:16.858397 14831 raft_consensus.cc:482] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:05:16.858491 14831 raft_consensus.cc:3037] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:05:16.860680 14831 raft_consensus.cc:504] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.861241 14831 leader_election.cc:288] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [CANDIDATE]: Term 1 election: Requested vote from peers 2a921a4ed67849e28a9f2acf723d4fc9 (127.3.240.195:33933), 2e66f6e0e88547b395c5d2165f3c20ff (127.3.240.193:36885)
I20260927 23:05:16.861584 14762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6419d711102f4f40ba24d82f074909b7" candidate_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a921a4ed67849e28a9f2acf723d4fc9"
I20260927 23:05:16.861768 14762 raft_consensus.cc:3037] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:05:16.861797 14495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6419d711102f4f40ba24d82f074909b7" candidate_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e66f6e0e88547b395c5d2165f3c20ff"
I20260927 23:05:16.861995 14495 raft_consensus.cc:3037] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:05:16.863921 14762 raft_consensus.cc:2454] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a515794a0c49b9a53d5658d5a2ffd1 in term 1.
I20260927 23:05:16.864270 14495 raft_consensus.cc:2454] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a515794a0c49b9a53d5658d5a2ffd1 in term 1.
I20260927 23:05:16.864409 14563 leader_election.cc:302] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a921a4ed67849e28a9f2acf723d4fc9, f4a515794a0c49b9a53d5658d5a2ffd1; no voters:
I20260927 23:05:16.864730 14831 raft_consensus.cc:2781] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:05:16.865168 14831 raft_consensus.cc:686] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 1 LEADER]: Becoming Leader. State: Replica: f4a515794a0c49b9a53d5658d5a2ffd1, State: Running, Role: LEADER
I20260927 23:05:16.865585 14831 consensus_queue.cc:237] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } }
I20260927 23:05:16.869865 14351 catalog_manager.cc:5495] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4a515794a0c49b9a53d5658d5a2ffd1 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4a515794a0c49b9a53d5658d5a2ffd1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 } health_report { overall_health: UNKNOWN } } }
I20260927 23:05:16.957720 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.953039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36744 (local address 127.3.240.194:43259)
0927 23:05:16.953214 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.953218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:16.953258 (+ 40us) server_negotiation.cc:407] Connection header received
0927 23:05:16.953443 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:16.953446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:16.953565 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:16.953690 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:05:16.954036 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.954740 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.956035 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:16.956396 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:16.956482 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:16.956830 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:05:16.957281 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:05:16.957406 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:16.957473 (+ 67us) server_negotiation.cc:299] Negotiation successful
0927 23:05:16.957527 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:05:16.965447 14762 raft_consensus.cc:1261] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 1 FOLLOWER]: Refusing update from remote peer f4a515794a0c49b9a53d5658d5a2ffd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:05:16.965443 14495 raft_consensus.cc:1261] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 1 FOLLOWER]: Refusing update from remote peer f4a515794a0c49b9a53d5658d5a2ffd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:05:16.966460 14836 consensus_queue.cc:1035] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e66f6e0e88547b395c5d2165f3c20ff" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:05:16.966785 14831 consensus_queue.cc:1035] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a921a4ed67849e28a9f2acf723d4fc9" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:05:16.975584 14839 mvcc.cc:205] Tried to move back new op lower bound from 7334094098280734720 to 7334094097884774400. Current Snapshot: MvccSnapshot[applied={T|T < 7334094098280734720}]
I20260927 23:05:16.978683 14840 mvcc.cc:205] Tried to move back new op lower bound from 7334094098280734720 to 7334094097884774400. Current Snapshot: MvccSnapshot[applied={T|T < 7334094098280734720}]
I20260927 23:05:16.980124 14842 mvcc.cc:205] Tried to move back new op lower bound from 7334094098280734720 to 7334094097884774400. Current Snapshot: MvccSnapshot[applied={T|T < 7334094098280734720}]
I20260927 23:05:16.990608 14350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:05:16.990923 14350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:05:16.994400 14350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dd00b263eba4b54b3f4786843c0ff21: Sending DeleteTablet for 3 replicas of tablet 6419d711102f4f40ba24d82f074909b7
I20260927 23:05:16.995496 14609 tablet_service.cc:1514] Processing DeleteTablet for tablet 6419d711102f4f40ba24d82f074909b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:05:16 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36738
I20260927 23:05:16.995546 14475 tablet_service.cc:1514] Processing DeleteTablet for tablet 6419d711102f4f40ba24d82f074909b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:05:16 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:40194
I20260927 23:05:16.996212 14742 tablet_service.cc:1514] Processing DeleteTablet for tablet 6419d711102f4f40ba24d82f074909b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:05:16 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:48156
I20260927 23:05:16.996501 14849 tablet_replica.cc:330] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: stopping tablet replica
I20260927 23:05:16.996901 14849 raft_consensus.cc:2229] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:05:16.997128 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 14412
I20260927 23:05:16.997447 14849 raft_consensus.cc:2258] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:05:16.999806 14849 ts_tablet_manager.cc:1905] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:05:17.004016 14848 tablet_replica.cc:330] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: stopping tablet replica
I20260927 23:05:17.004964 14850 tablet_replica.cc:330] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: stopping tablet replica
I20260927 23:05:17.005278 14850 raft_consensus.cc:2229] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:05:17.005707 14850 raft_consensus.cc:2258] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:05:17.004458 14848 raft_consensus.cc:2229] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:05:17.006661 14848 raft_consensus.cc:2258] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:05:17.008605 14848 ts_tablet_manager.cc:1905] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:05:17.009223 14849 ts_tablet_manager.cc:1918] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:05:17.009416 14849 log.cc:1196] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/6419d711102f4f40ba24d82f074909b7
I20260927 23:05:17.009936 14849 ts_tablet_manager.cc:1939] T 6419d711102f4f40ba24d82f074909b7 P 2e66f6e0e88547b395c5d2165f3c20ff: Deleting consensus metadata
I20260927 23:05:17.009956 14850 ts_tablet_manager.cc:1905] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:05:17.011562 14337 catalog_manager.cc:4841] TS 2e66f6e0e88547b395c5d2165f3c20ff (127.3.240.193:36885): tablet 6419d711102f4f40ba24d82f074909b7 (table test-table [id=42410ab4607f4d8bb64937db50e90484]) successfully deleted
I20260927 23:05:17.017129 14848 ts_tablet_manager.cc:1918] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:05:17.017154 14850 ts_tablet_manager.cc:1918] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:05:17.017283 14850 log.cc:1196] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/6419d711102f4f40ba24d82f074909b7
I20260927 23:05:17.017305 14848 log.cc:1196] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/6419d711102f4f40ba24d82f074909b7
I20260927 23:05:17.017781 14850 ts_tablet_manager.cc:1939] T 6419d711102f4f40ba24d82f074909b7 P 2a921a4ed67849e28a9f2acf723d4fc9: Deleting consensus metadata
I20260927 23:05:17.017809 14848 ts_tablet_manager.cc:1939] T 6419d711102f4f40ba24d82f074909b7 P f4a515794a0c49b9a53d5658d5a2ffd1: Deleting consensus metadata
I20260927 23:05:17.019270 14336 catalog_manager.cc:4841] TS 2a921a4ed67849e28a9f2acf723d4fc9 (127.3.240.195:33933): tablet 6419d711102f4f40ba24d82f074909b7 (table test-table [id=42410ab4607f4d8bb64937db50e90484]) successfully deleted
I20260927 23:05:17.019516 14338 catalog_manager.cc:4841] TS f4a515794a0c49b9a53d5658d5a2ffd1 (127.3.240.194:43259): tablet 6419d711102f4f40ba24d82f074909b7 (table test-table [id=42410ab4607f4d8bb64937db50e90484]) successfully deleted
Sep 27 23:05:17 dist-test-slave-lmfp krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:05:17.028709 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:16.998836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.3.240.193:36885)
0927 23:05:16.999008 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:16.999014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:17.013420 (+ 14406us) server_negotiation.cc:407] Connection header received
0927 23:05:17.013595 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:17.013600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:17.013683 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:17.013779 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:17.015284 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:17.016324 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:17.018321 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:17.018863 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:17.018955 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:17.023273 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:17.023284 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:17.023296 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:17.023326 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:17.026059 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:17.026698 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:17.026704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:17.026707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:17.026829 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:17.027231 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:17.027239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:17.027242 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:17.027572 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:17.027854 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:17.028178 (+ 324us) server_negotiation.cc:299] Negotiation successful
0927 23:05:17.028394 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 23:05:17.029430 14535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:17.115729 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 14412
I20260927 23:05:17.133281 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 14546
Sep 27 23:05:17 dist-test-slave-lmfp krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:05:17.148465 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:17.134329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.3.240.194:43259)
0927 23:05:17.134552 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:17.134557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:17.134572 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:05:17.134622 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:17.134626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:17.134723 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:17.134859 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:17.136132 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:17.136980 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:17.138153 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:17.138575 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:17.138726 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:17.142696 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:17.142712 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:17.142730 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:17.142763 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:17.145822 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:17.146469 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:17.146477 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:17.146480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:17.146602 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:17.146959 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:17.146965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:17.146968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:17.147354 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:17.147681 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:17.147906 (+ 225us) server_negotiation.cc:299] Negotiation successful
0927 23:05:17.148115 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 23:05:17.149274 14669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:17.259110 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 14546
I20260927 23:05:17.276808 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 14679
Sep 27 23:05:17 dist-test-slave-lmfp krb5kdc[14306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:05:17.290334 14822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:17.277493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.3.240.195:33933)
0927 23:05:17.277643 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:17.277648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:17.277664 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:05:17.277827 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:17.277831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:17.277891 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:17.277972 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:17.279202 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:17.279883 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:17.281031 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:17.281408 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:17.281494 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:17.285424 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:17.285436 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:17.285452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:17.285485 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:17.288160 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:17.288811 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:17.288818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:17.288820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:17.288941 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:17.289212 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:17.289215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:17.289217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:17.289517 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:17.289745 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:17.289942 (+ 197us) server_negotiation.cc:299] Negotiation successful
0927 23:05:17.290080 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 23:05:17.290922 14802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:17.376919 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 14679
I20260927 23:05:17.392979 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 14319
I20260927 23:05:17.394043 14381 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:05:17.497116 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 14319
2026-09-27T23:05:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4266 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:05:17 dist-test-slave-lmfp krb5kdc[14858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:05:17 dist-test-slave-lmfp krb5kdc[14858](info): set up 2 sockets
Sep 27 23:05:17 dist-test-slave-lmfp krb5kdc[14858](info): commencing operation
krb5kdc: starting...
W20260927 23:05:18.556028 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:05:18 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:05:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:05:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/rangeradmin_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/rangeradmin_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/rangerlookup_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/rangerlookup_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
I20260927 23:05:18.842267 4035 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9OvN8C/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/postgres -l logfile start
2026-09-27 23:05:21.100 UTC [14888] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 23:05:21.100 UTC [14888] LOG: listening on IPv4 address "127.3.240.212", port 37371
2026-09-27 23:05:21.104 UTC [14888] LOG: listening on Unix socket "/tmp/.s.PGSQL.37371"
2026-09-27 23:05:21.127 UTC [14891] LOG: database system was shut down at 2026-09-27 23:05:19 UTC
2026-09-27 23:05:21.131 UTC [14888] LOG: database system is ready to accept connections
I20260927 23:05:22.071767 4035 mini_postgres.cc:96] Postgres bound to 37371
2026-09-27 23:05:22.088 UTC [14899] FATAL: database "slave" does not exist
127.3.240.212:37371 - accepting connections
I20260927 23:05:22.089031 4035 mini_ranger.cc:161] Starting Ranger...
I20260927 23:05:22.111289 4035 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 23:05:22.524796 4035 mini_ranger.cc:87] Created ranger Postgres database
I20260927 23:05:22.524923 4035 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-admin
2026-09-27 23:05:22,791 [I] DB FLAVOR :POSTGRES
2026-09-27 23:05:22,792 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 23:05:22,792 [I] Checking connection..
2026-09-27 23:05:22,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 23:05:23,020 [I] Checking connection passed.
2026-09-27 23:05:23,020 [I] --------- Verifying version history table ---------
2026-09-27 23:05:23,020 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 23:05:23,270 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 23:05:23,270 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 23:05:23,491 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 23:05:23,491 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 23:05:23,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 23:05:23.725 UTC [14963] WARNING: there is no transaction in progress
2026-09-27 23:05:23,731 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 23:05:23,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 23:05:23,971 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 23:05:23,971 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 23:05:23,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 23:05:24,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 23:05:24,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lmfp.c.gcp-upstream.internal','N') ;"
2026-09-27 23:05:24,667 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 23:05:24,887 [I] Table x_portal_user does not exist in database ranger
2026-09-27 23:05:24,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 23:05:25,134 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 23:05:25,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 23:05:25,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 23:05:25,607 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 23:05:25,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 23:05:26.489 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.503 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.517 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.531 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.548 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.567 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.629 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.635 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.648 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.657 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.666 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.678 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.687 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.693 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.699 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:26.705 UTC [15071] WARNING: there is no transaction in progress
2026-09-27 23:05:27,249 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 23:05:27,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 23:05:27,496 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 23:05:27,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 23:05:27,745 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 23:05:27,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 23:05:28,002 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 23:05:28,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 23:05:28,247 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 23:05:28,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lmfp.c.gcp-upstream.internal';"
2026-09-27 23:05:28,739 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 23:05:28,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 23:05:28,969 [I] DB_PATCHES have already been applied
I20260927 23:05:28.974782 4035 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-admin:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9OvN8C/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 23:05:28.974895 4035 mini_ranger.cc:193] Using host: 127.3.240.212
I20260927 23:05:28.989370 4035 mini_ranger.cc:227] Ranger admin URL: http://127.3.240.212:58795
Sep 27, 2026 11:05:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:05:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:05:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 11:05:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 11:05:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 11:05:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 11:05:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 23:05:30.098670 4035 mini_ranger.cc:160] Time spent starting Ranger: real 8.010s user 0.003s sys 0.032s
Sep 27, 2026 11:05:48 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 11:05:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 11:05:48 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 11:05:48 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 11:05:49 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 23:05:50.024487 4035 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/rangerkms_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/rangerkms_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.240.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.240.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 23:05:50 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 23:05:50.180802 4035 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 23:05:50.200420 4035 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 23:05:51.092466 4035 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 23:05:51.092644 4035 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms
I20260927 23:05:51.092809 4035 mini_ranger_kms.cc:227] Using postgres at 127.3.240.212:37371
2026-09-27 23:05:51,508 [I] DB FLAVOR :POSTGRES
2026-09-27 23:05:51,509 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 23:05:51,509 [I] Checking connection
2026-09-27 23:05:51,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 23:05:51,772 [I] connection success
2026-09-27 23:05:51,772 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 23:05:51,772 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 23:05:51,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 23:05:52,013 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 23:05:52,013 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 23:05:52,013 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 23:05:52,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37371/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 23:05:52,286 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 23:05:52.710835 4035 mini_ranger_kms.cc:310] Created kms service
I20260927 23:05:52.900367 4035 mini_ranger_kms.cc:326] Created kudu user
I20260927 23:05:52.945148 4035 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 23:05:52 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550350, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:05:53.343981 4035 mini_ranger_kms.cc:384] Added ranger policy
I20260927 23:05:53.344277 4035 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms:/tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 23:05:53.344349 4035 mini_ranger_kms.cc:240] Using host: 127.3.240.212
I20260927 23:05:53.364907 4035 mini_ranger_kms.cc:276] Ranger KMS PID: 15251
I20260927 23:05:53.365072 4035 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.240.212:39333
Sep 27, 2026 11:05:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:05:54 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:05:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 11:05:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 11:05:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 11:05:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260927 23:05:55.606597 4035 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.426s user 0.113s sys 0.312s
I20260927 23:05:55.606844 4035 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 23:05:55.607110 4035 mini_ranger_kms.cc:401] 127.3.240.212:39333/kms/v1/keys
Sep 27 23:05:55 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790550355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 27 23:05:56 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Sep 27 23:05:56 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550355, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Sep 27 23:05:58 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:05:58.916529 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42513
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:42513
--ranger_config_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:39333/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:05:59.247226 15302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:05:59.247539 15302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:05:59.247690 15302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:05:59.254496 15302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:05:59.254631 15302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:05:59.254731 15302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:05:59.254828 15302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:05:59.254910 15302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:05:59.264444 15302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.240.212:39333/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:42513
--ranger_config_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:05:59.265622 15302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:05:59.266829 15302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:05:59.278750 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:59.278795 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:05:59.278817 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:05:59.279100 15302 server_base.cc:1034] running on GCE node
I20260927 23:05:59.279814 15302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:05:59.281044 15302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:05:59.282267 15302 hybrid_clock.cc:648] HybridClock initialized: now 1790550359282238 us; error 45 us; skew 500 ppm
I20260927 23:05:59.282796 15302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:05:59 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:05:59.291065 15302 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:05:59.292495 15302 webserver.cc:460] Webserver started at http://127.3.240.254:36433/ using document root <none> and password file <none>
I20260927 23:05:59.293113 15302 fs_manager.cc:359] Metadata directory not provided
I20260927 23:05:59.293241 15302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:05:59.293555 15302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 23:05:59 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550359, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:05:59.353946 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "5f7f292167394378ae972ceba6c98983"
format_stamp: "Formatted at 2026-09-27 23:05:59 on dist-test-slave-lmfp"
server_key: "b945cf92e84311354764a3a66c3cf29c"
server_key_iv: "6032b51bac80414d13b99dcaa9517e1b"
server_key_version: "kuduclusterkey@0"
I20260927 23:05:59.354830 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "5f7f292167394378ae972ceba6c98983"
format_stamp: "Formatted at 2026-09-27 23:05:59 on dist-test-slave-lmfp"
server_key: "b945cf92e84311354764a3a66c3cf29c"
server_key_iv: "6032b51bac80414d13b99dcaa9517e1b"
server_key_version: "kuduclusterkey@0"
I20260927 23:05:59.360463 15302 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 23:05:59.377198 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:05:59.378705 15302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:05:59.378873 15302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "5f7f292167394378ae972ceba6c98983"
format_stamp: "Formatted at 2026-09-27 23:05:59 on dist-test-slave-lmfp"
server_key: "b945cf92e84311354764a3a66c3cf29c"
server_key_iv: "6032b51bac80414d13b99dcaa9517e1b"
server_key_version: "kuduclusterkey@0"
I20260927 23:05:59.379166 15302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:05:59.401059 15302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:05:59.405368 15302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:05:59.405730 15302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:05:59.426450 15302 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:42513
I20260927 23:05:59.426543 15368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:42513 every 8 connection(s)
I20260927 23:05:59.427845 15302 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:05:59.429288 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 15302
I20260927 23:05:59.429533 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:05:59.430469 4035 external_mini_cluster.cc:1428] Setting key 936fe5b8c2693b1f6d4e898c4616d8b6
I20260927 23:05:59.438337 15369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 23:05:59 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:05:59.453322 15369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983: Bootstrap starting.
I20260927 23:05:59.457340 15369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983: Neither blocks nor log segments found. Creating new log.
I20260927 23:05:59.458451 15369 log.cc:824] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983: Log is configured to *not* fsync() on all Append() calls
I20260927 23:05:59.459699 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:05:59.433011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.3.240.254:42513)
0927 23:05:59.440620 (+ 7609us) server_negotiation.cc:206] Beginning negotiation
0927 23:05:59.440635 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:05:59.440682 (+ 47us) server_negotiation.cc:407] Connection header received
0927 23:05:59.443684 (+ 3002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:05:59.443704 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:05:59.444055 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:05:59.444422 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:05:59.445632 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:59.446930 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:59.448168 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:05:59.448750 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:05:59.448925 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:05:59.452498 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:05:59.452505 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:05:59.452517 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:05:59.452551 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:05:59.455389 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:59.456014 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:59.456020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:59.456025 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:59.456183 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:05:59.456556 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:05:59.456562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:05:59.456565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:05:59.457008 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:05:59.457284 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 23:05:59.458140 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 23:05:59.458419 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7383,"thread_start_us":7156,"threads_started":1}
I20260927 23:05:59.462471 15369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983: No bootstrap required, opened a new log
I20260927 23:05:59.469551 15369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7f292167394378ae972ceba6c98983" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42513 } }
I20260927 23:05:59.469848 15369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:05:59.469964 15369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f7f292167394378ae972ceba6c98983, State: Initialized, Role: FOLLOWER
I20260927 23:05:59.470624 15369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7f292167394378ae972ceba6c98983" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42513 } }
I20260927 23:05:59.470887 15369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:05:59.471060 15369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:05:59.471220 15369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:05:59.474001 15369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7f292167394378ae972ceba6c98983" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42513 } }
I20260927 23:05:59.474471 15369 leader_election.cc:302] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f7f292167394378ae972ceba6c98983; no voters:
I20260927 23:05:59.475131 15369 leader_election.cc:288] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:05:59.475260 15374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:05:59.479578 15374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [term 1 LEADER]: Becoming Leader. State: Replica: 5f7f292167394378ae972ceba6c98983, State: Running, Role: LEADER
I20260927 23:05:59.480221 15374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7f292167394378ae972ceba6c98983" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42513 } }
I20260927 23:05:59.480446 15369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:05:59.490660 15375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f7f292167394378ae972ceba6c98983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7f292167394378ae972ceba6c98983" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42513 } } }
I20260927 23:05:59.490981 15375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [sys.catalog]: This master's current role is: LEADER
I20260927 23:05:59.497291 15376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f7f292167394378ae972ceba6c98983. Latest consensus state: current_term: 1 leader_uuid: "5f7f292167394378ae972ceba6c98983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f7f292167394378ae972ceba6c98983" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42513 } } }
I20260927 23:05:59.498102 15376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983 [sys.catalog]: This master's current role is: LEADER
I20260927 23:05:59.499636 15385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:05:59.507414 15385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:05:59.524036 15369 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 23:05:59.525584 15385 catalog_manager.cc:1347] Generated new cluster ID: e06b20f5e72d474f8bdec407c5e50781
I20260927 23:05:59.525831 15385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:05:59.556217 15385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:05:59.558560 15385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:05:59.567953 15385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983: Generated new TSK 0
I20260927 23:05:59.569057 15385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:06:00.270887 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:39333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42513
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:06:00 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790550360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 23:06:00.713415 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:00.713769 15415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:00.713966 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:00.725332 15415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:00.725517 15415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:00.725726 15415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:06:00.741043 15415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.240.212:39333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:00.742615 15415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:00.744237 15415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:00.762125 15426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:00.763386 15425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:00.775610 15415 server_base.cc:1034] running on GCE node
W20260927 23:06:00.776474 15428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:00.777418 15415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:00.778765 15415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:00.780049 15415 hybrid_clock.cc:648] HybridClock initialized: now 1790550360779964 us; error 73 us; skew 500 ppm
I20260927 23:06:00.780480 15415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:00 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:00.784710 15415 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:06:00.798137 15415 webserver.cc:460] Webserver started at http://127.3.240.193:34729/ using document root <none> and password file <none>
I20260927 23:06:00.799348 15415 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:00.799640 15415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:00.800073 15415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 23:06:00 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:00.819674 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "b1228ba1689c4f4d99359e2ba47259b0"
format_stamp: "Formatted at 2026-09-27 23:06:00 on dist-test-slave-lmfp"
server_key: "a6f608201cf65b230bd618328b92953f"
server_key_iv: "b3fb324731eafbbe58acd8e00d0e1efd"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:00.820853 15415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "b1228ba1689c4f4d99359e2ba47259b0"
format_stamp: "Formatted at 2026-09-27 23:06:00 on dist-test-slave-lmfp"
server_key: "a6f608201cf65b230bd618328b92953f"
server_key_iv: "b3fb324731eafbbe58acd8e00d0e1efd"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:00.829600 15415 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260927 23:06:00.867645 15435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:00.870518 15415 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.007s
I20260927 23:06:00.870710 15415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "b1228ba1689c4f4d99359e2ba47259b0"
format_stamp: "Formatted at 2026-09-27 23:06:00 on dist-test-slave-lmfp"
server_key: "a6f608201cf65b230bd618328b92953f"
server_key_iv: "b3fb324731eafbbe58acd8e00d0e1efd"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:00.870976 15415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:00.918514 15415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:00.924162 15415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:00.924546 15415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:00.925707 15415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:00.927452 15415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:00.927534 15415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:00.927685 15415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:00.927742 15415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 27 23:06:01 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:01.061211 15415 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:45875
I20260927 23:06:01.061231 15547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:45875 every 8 connection(s)
I20260927 23:06:01.063045 15415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:06:01.068789 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 15415
I20260927 23:06:01.068971 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:06:01.069758 4035 external_mini_cluster.cc:1428] Setting key 8cdc220a36dc710921fc3218a1b8bf15
Sep 27 23:06:01 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260927 23:06:01.170590 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:01.067344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:46177 (local address 127.3.240.254:42513)
0927 23:06:01.123409 (+ 56065us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:01.123415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:01.123443 (+ 28us) server_negotiation.cc:407] Connection header received
0927 23:06:01.127522 (+ 4079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:01.127528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:01.127632 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:01.127744 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:01.137892 (+ 10148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:01.139250 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:01.148660 (+ 9410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:01.149753 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:01.149864 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:01.159637 (+ 9773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:01.159645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:01.159650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:01.159690 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:01.163409 (+ 3719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:01.167936 (+ 4527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:01.167942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:01.167945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:01.168118 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:01.168628 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:01.168633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:01.168636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:01.168939 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:01.169067 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:01.169997 (+ 930us) server_negotiation.cc:299] Negotiation successful
0927 23:06:01.170197 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55924,"thread_start_us":59157,"threads_started":1}
I20260927 23:06:01.172812 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:01.068010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42513 (local address 127.3.240.193:46177)
0927 23:06:01.116401 (+ 48391us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:01.116465 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:01.127015 (+ 10550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:01.128231 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:01.128243 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:01.128879 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:01.129733 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:01.129755 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:01.139500 (+ 9745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:01.139506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:01.141552 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:01.141576 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:01.151004 (+ 9428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:01.151013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:01.151334 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:01.152239 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:01.152269 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:01.159355 (+ 7086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:01.167160 (+ 7805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:01.167170 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:01.167189 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:01.167783 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:01.168265 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:01.168270 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:01.168273 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:01.168504 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:01.169155 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:01.169162 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:01.169589 (+ 427us) client_negotiation.cc:769] Sending connection context
0927 23:06:01.170799 (+ 1210us) client_negotiation.cc:240] Negotiation successful
0927 23:06:01.171376 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48062,"thread_start_us":46881,"threads_started":1}
I20260927 23:06:01.174896 15548 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42513
I20260927 23:06:01.175248 15548 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:01.176087 15548 heartbeater.cc:511] Master 127.3.240.254:42513 requested a full tablet report, sending...
I20260927 23:06:01.178570 15334 ts_manager.cc:194] Registered new tserver with Master: b1228ba1689c4f4d99359e2ba47259b0 (127.3.240.193:45875)
I20260927 23:06:01.180552 15334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:46177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
W20260927 23:06:01.201272 4035 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.3.240.194: real 0.129s user 0.005s sys 0.010s
I20260927 23:06:01.202153 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:39333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42513
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:06:01 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550360, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
W20260927 23:06:01.640599 15554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:01.640869 15554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:01.641026 15554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:01.648139 15554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:01.648226 15554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:01.648447 15554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:06:01.657358 15554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.240.212:39333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:01.658484 15554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:01.659782 15554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:01.672825 15568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:01.672863 15571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:01.672883 15569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:01.673418 15554 server_base.cc:1034] running on GCE node
I20260927 23:06:01.674052 15554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:01.674958 15554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:01.676208 15554 hybrid_clock.cc:648] HybridClock initialized: now 1790550361676156 us; error 46 us; skew 500 ppm
I20260927 23:06:01.676546 15554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:01 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550361, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:01.679708 15554 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:06:01.681133 15554 webserver.cc:460] Webserver started at http://127.3.240.194:33491/ using document root <none> and password file <none>
I20260927 23:06:01.681797 15554 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:01.681929 15554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:01.682202 15554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 23:06:01 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550361, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:01.697214 15554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "3bd8f9f00fd34640b56a009dd9a78a12"
format_stamp: "Formatted at 2026-09-27 23:06:01 on dist-test-slave-lmfp"
server_key: "631001c1b8c6c92840723b0da6a1982e"
server_key_iv: "b7c928c59b81bcc6691104ce4435f916"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:01.698107 15554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "3bd8f9f00fd34640b56a009dd9a78a12"
format_stamp: "Formatted at 2026-09-27 23:06:01 on dist-test-slave-lmfp"
server_key: "631001c1b8c6c92840723b0da6a1982e"
server_key_iv: "b7c928c59b81bcc6691104ce4435f916"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:01.703876 15554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 23:06:01.718856 15578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:01.720403 15554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:06:01.720588 15554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "3bd8f9f00fd34640b56a009dd9a78a12"
format_stamp: "Formatted at 2026-09-27 23:06:01 on dist-test-slave-lmfp"
server_key: "631001c1b8c6c92840723b0da6a1982e"
server_key_iv: "b7c928c59b81bcc6691104ce4435f916"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:01.720824 15554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:01.741235 15554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:01.745009 15554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:01.745296 15554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:01.746567 15554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:01.748037 15554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:01.748152 15554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:01.748297 15554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:01.748371 15554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:01.782960 15554 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:46089
I20260927 23:06:01.783036 15690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:46089 every 8 connection(s)
I20260927 23:06:01.784282 15554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:06:01.791428 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 15554
I20260927 23:06:01.791615 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:06:01.792090 4035 external_mini_cluster.cc:1428] Setting key 493a2beb92ece3026a5811278c8bb204
Sep 27 23:06:01 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550361, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:01.805132 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:01.786726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:39235 (local address 127.3.240.254:42513)
0927 23:06:01.787231 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:01.787237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:01.788184 (+ 947us) server_negotiation.cc:407] Connection header received
0927 23:06:01.790255 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:01.790260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:01.790323 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:01.790418 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:01.791992 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:01.792940 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:01.794560 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:01.795264 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:01.795423 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:01.798670 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:01.798678 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:01.798685 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:01.798728 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:01.801715 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:01.802726 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:01.802730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:01.802732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:01.802860 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:01.803389 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:01.803392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:01.803394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:01.803671 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:01.803776 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:01.804632 (+ 856us) server_negotiation.cc:299] Negotiation successful
0927 23:06:01.804788 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":214,"threads_started":1}
I20260927 23:06:01.806526 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:01.787331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42513 (local address 127.3.240.194:39235)
0927 23:06:01.788026 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:01.788062 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:01.790006 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:01.790666 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:01.790674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:01.791180 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:01.791835 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:01.791854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:01.793093 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:01.793098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:01.794412 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:01.794430 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:01.795510 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:01.795516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:01.795726 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:01.796417 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:01.796439 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:01.798470 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:01.801859 (+ 3389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:01.801869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:01.801887 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:01.802597 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:01.802975 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:01.802983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:01.803011 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:01.803277 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:01.803799 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:01.803809 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:01.804245 (+ 436us) client_negotiation.cc:769] Sending connection context
0927 23:06:01.804537 (+ 292us) client_negotiation.cc:240] Negotiation successful
0927 23:06:01.805167 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":297,"threads_started":1}
I20260927 23:06:01.807977 15691 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42513
I20260927 23:06:01.808301 15691 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:01.809037 15691 heartbeater.cc:511] Master 127.3.240.254:42513 requested a full tablet report, sending...
I20260927 23:06:01.810539 15334 ts_manager.cc:194] Registered new tserver with Master: 3bd8f9f00fd34640b56a009dd9a78a12 (127.3.240.194:46089)
I20260927 23:06:01.811564 15334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:39235
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:06:01.844450 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:39333/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42513
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:06:02.168346 15697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:02.168630 15697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:02.168826 15697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:02.176062 15697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:02.176216 15697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:02.176435 15697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:06:02.184226 15548 heartbeater.cc:503] Master 127.3.240.254:42513 was elected leader, sending a full tablet report...
I20260927 23:06:02.186745 15697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.240.212:39333/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:02.188050 15697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:02.189307 15697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:02.202585 15705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:02.202553 15703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:02.202950 15702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:02.203392 15697 server_base.cc:1034] running on GCE node
I20260927 23:06:02.204010 15697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:02.204857 15697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:02.206122 15697 hybrid_clock.cc:648] HybridClock initialized: now 1790550362206054 us; error 45 us; skew 500 ppm
I20260927 23:06:02.206485 15697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:02 dist-test-slave-lmfp krb5kdc[14858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550362, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:02.209942 15697 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:06:02.211510 15697 webserver.cc:460] Webserver started at http://127.3.240.195:39957/ using document root <none> and password file <none>
I20260927 23:06:02.212186 15697 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:02.212333 15697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:02.212656 15697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 23:06:02 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550362, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:02.229714 15697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73"
format_stamp: "Formatted at 2026-09-27 23:06:02 on dist-test-slave-lmfp"
server_key: "d4ff7e32b55bedc39681cfbf95d7e36c"
server_key_iv: "9bb2da912357aa66ca8bb4adb610bd7e"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:02.230763 15697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73"
format_stamp: "Formatted at 2026-09-27 23:06:02 on dist-test-slave-lmfp"
server_key: "d4ff7e32b55bedc39681cfbf95d7e36c"
server_key_iv: "9bb2da912357aa66ca8bb4adb610bd7e"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:02.237633 15697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260927 23:06:02.252745 15712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:02.254395 15697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 23:06:02.254578 15697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73"
format_stamp: "Formatted at 2026-09-27 23:06:02 on dist-test-slave-lmfp"
server_key: "d4ff7e32b55bedc39681cfbf95d7e36c"
server_key_iv: "9bb2da912357aa66ca8bb4adb610bd7e"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:02.254803 15697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:02.278849 15697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:02.283033 15697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:02.283342 15697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:02.284343 15697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:02.285633 15697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:02.285701 15697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:02.285826 15697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:02.285915 15697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:02.325923 15697 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:37643
I20260927 23:06:02.325953 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:37643 every 8 connection(s)
I20260927 23:06:02.327179 15697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:06:02.335366 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 15697
I20260927 23:06:02.335552 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:06:02.335983 4035 external_mini_cluster.cc:1428] Setting key fed554189f71c7e9bcabe595bffdc946
Sep 27 23:06:02 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550362, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:02.346032 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.329669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:41839 (local address 127.3.240.254:42513)
0927 23:06:02.330110 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.330115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.330914 (+ 799us) server_negotiation.cc:407] Connection header received
0927 23:06:02.332678 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.332684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.332783 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.332899 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:02.334217 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.334912 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.336638 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.337203 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.337339 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.340638 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:02.340642 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:02.340646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:02.340675 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:02.343617 (+ 2942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:02.344243 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:02.344246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:02.344249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:02.344356 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:02.344705 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:02.344708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:02.344710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:02.344948 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:02.345052 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.345639 (+ 587us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.345784 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":213,"threads_started":1}
I20260927 23:06:02.346949 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.330094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42513 (local address 127.3.240.195:41839)
0927 23:06:02.330764 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:02.330800 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:02.332411 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:02.333124 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:02.333132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:02.333517 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:02.334057 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.334073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.335131 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.335138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.336486 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.336497 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.337432 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.337439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.337652 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.338350 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:02.338374 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:02.340422 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:02.343736 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:02.343743 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:02.343754 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:02.344150 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:02.344442 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:02.344445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:02.344447 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:02.344621 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:02.345040 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:02.345045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:02.345356 (+ 311us) client_negotiation.cc:769] Sending connection context
0927 23:06:02.345568 (+ 212us) client_negotiation.cc:240] Negotiation successful
0927 23:06:02.345923 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":256,"threads_started":1}
I20260927 23:06:02.348253 15825 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42513
I20260927 23:06:02.348531 15825 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:02.349196 15825 heartbeater.cc:511] Master 127.3.240.254:42513 requested a full tablet report, sending...
I20260927 23:06:02.350526 15334 ts_manager.cc:194] Registered new tserver with Master: e653c2a6aa9d4c72a1bdda8a4a9f4c73 (127.3.240.195:37643)
I20260927 23:06:02.351346 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:06:02.351593 15334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:41839
I20260927 23:06:02.367708 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.355488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.3.240.254:42513)
0927 23:06:02.355677 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.355685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.356025 (+ 340us) server_negotiation.cc:407] Connection header received
0927 23:06:02.356159 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.356163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.356225 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.356293 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:02.357821 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.358702 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.360007 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.360473 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.360563 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.362708 (+ 2145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:02.362712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:02.362715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:02.362740 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:02.365264 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:02.365794 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:02.365797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:02.365799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:02.365920 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:02.366201 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:02.366203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:02.366205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:02.366407 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:02.366561 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.367245 (+ 684us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.367392 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:06:02.371250 15334 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:06:02.378043 15334 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:06:02.400847 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.389724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46089 (local address 127.0.0.1:55520)
0927 23:06:02.390846 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:02.390890 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:02.391137 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:02.392506 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:02.392516 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:02.392557 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:02.393121 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.393135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.394973 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.394980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.397337 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.397355 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.399812 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.399819 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.400026 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.400079 (+ 53us) client_negotiation.cc:769] Sending connection context
0927 23:06:02.400292 (+ 213us) client_negotiation.cc:240] Negotiation successful
0927 23:06:02.400464 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":3648,"threads_started":1}
I20260927 23:06:02.401791 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.389888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.3.240.194:46089)
0927 23:06:02.391789 (+ 1901us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.391801 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.391835 (+ 34us) server_negotiation.cc:407] Connection header received
0927 23:06:02.391944 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.391955 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.392261 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.392472 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:02.393305 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.394828 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.397520 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.399341 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.399479 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.399682 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.401228 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.401391 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1693,"thread_start_us":1588,"threads_started":1}
I20260927 23:06:02.402191 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.389961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:37643 (local address 127.0.0.1:41018)
0927 23:06:02.393663 (+ 3702us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:02.393680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:02.393785 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:02.394579 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:02.394583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:02.394603 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:02.394952 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.394963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.396833 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.396838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.399422 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.399437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.401407 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.401413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.401570 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.401604 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 23:06:02.401696 (+ 92us) client_negotiation.cc:240] Negotiation successful
0927 23:06:02.401821 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3562,"thread_start_us":939,"threads_started":1}
I20260927 23:06:02.402513 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.390615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.3.240.195:37643)
0927 23:06:02.393309 (+ 2694us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.393321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.393889 (+ 568us) server_negotiation.cc:407] Connection header received
0927 23:06:02.393972 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.393979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.394179 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.394351 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:02.395161 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.396676 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.399577 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.400984 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.401099 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.401287 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.401998 (+ 711us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.402174 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2492,"thread_start_us":2285,"threads_started":1}
I20260927 23:06:02.405601 15626 tablet_service.cc:1467] Processing CreateTablet for tablet 956c56e248544c5e806e45d73d082ccd (DEFAULT_TABLE table=test-table [id=afbe214edeea4c60818ca8930af0ab85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:06:02.406085 15760 tablet_service.cc:1467] Processing CreateTablet for tablet 956c56e248544c5e806e45d73d082ccd (DEFAULT_TABLE table=test-table [id=afbe214edeea4c60818ca8930af0ab85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:06:02.407043 15626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956c56e248544c5e806e45d73d082ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:02.407390 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.393734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45875 (local address 127.0.0.1:36726)
0927 23:06:02.400949 (+ 7215us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:02.400970 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:02.401089 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:02.401736 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:02.401740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:02.401761 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:02.402166 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.402177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.403886 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.403895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.405367 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.405377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.406669 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.406674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.406828 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.406870 (+ 42us) client_negotiation.cc:769] Sending connection context
0927 23:06:02.406952 (+ 82us) client_negotiation.cc:240] Negotiation successful
0927 23:06:02.407077 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7104,"thread_start_us":6979,"threads_started":1}
I20260927 23:06:02.407459 15760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956c56e248544c5e806e45d73d082ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:02.408282 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.394106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.3.240.193:45875)
0927 23:06:02.400685 (+ 6579us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.400695 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.401209 (+ 514us) server_negotiation.cc:407] Connection header received
0927 23:06:02.401305 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.401313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.401526 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.402380 (+ 854us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:02.402590 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.403732 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.405507 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.406488 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.406623 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.406829 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.407959 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.408069 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6417,"thread_start_us":6258,"threads_started":1}
I20260927 23:06:02.410861 15483 tablet_service.cc:1467] Processing CreateTablet for tablet 956c56e248544c5e806e45d73d082ccd (DEFAULT_TABLE table=test-table [id=afbe214edeea4c60818ca8930af0ab85]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:06:02.412127 15483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956c56e248544c5e806e45d73d082ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:02.423242 15844 tablet_bootstrap.cc:492] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Bootstrap starting.
I20260927 23:06:02.423452 15846 tablet_bootstrap.cc:492] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Bootstrap starting.
I20260927 23:06:02.423476 15845 tablet_bootstrap.cc:492] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Bootstrap starting.
I20260927 23:06:02.427059 15846 tablet_bootstrap.cc:654] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:02.427229 15844 tablet_bootstrap.cc:654] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:02.428223 15846 log.cc:824] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:02.428320 15845 tablet_bootstrap.cc:654] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:02.429446 15845 log.cc:824] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:02.428463 15844 log.cc:824] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:02.439424 15845 tablet_bootstrap.cc:492] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: No bootstrap required, opened a new log
I20260927 23:06:02.439824 15845 ts_tablet_manager.cc:1397] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20260927 23:06:02.452688 15845 raft_consensus.cc:348] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.453296 15845 raft_consensus.cc:374] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:02.453500 15845 raft_consensus.cc:729] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e653c2a6aa9d4c72a1bdda8a4a9f4c73, State: Initialized, Role: FOLLOWER
I20260927 23:06:02.453891 15846 tablet_bootstrap.cc:492] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: No bootstrap required, opened a new log
I20260927 23:06:02.454250 15846 ts_tablet_manager.cc:1397] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.026s
I20260927 23:06:02.454309 15844 tablet_bootstrap.cc:492] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: No bootstrap required, opened a new log
I20260927 23:06:02.454237 15845 consensus_queue.cc:260] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.454627 15844 ts_tablet_manager.cc:1397] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.013s
I20260927 23:06:02.459120 15825 heartbeater.cc:503] Master 127.3.240.254:42513 was elected leader, sending a full tablet report...
I20260927 23:06:02.459672 15845 ts_tablet_manager.cc:1428] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20260927 23:06:02.465797 15846 raft_consensus.cc:348] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.465798 15844 raft_consensus.cc:348] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.466219 15844 raft_consensus.cc:374] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:02.466218 15846 raft_consensus.cc:374] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:02.466316 15846 raft_consensus.cc:729] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1228ba1689c4f4d99359e2ba47259b0, State: Initialized, Role: FOLLOWER
I20260927 23:06:02.466318 15844 raft_consensus.cc:729] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd8f9f00fd34640b56a009dd9a78a12, State: Initialized, Role: FOLLOWER
I20260927 23:06:02.466949 15846 consensus_queue.cc:260] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.466949 15844 consensus_queue.cc:260] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.468488 15691 heartbeater.cc:503] Master 127.3.240.254:42513 was elected leader, sending a full tablet report...
I20260927 23:06:02.468745 15846 ts_tablet_manager.cc:1428] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260927 23:06:02.468994 15844 ts_tablet_manager.cc:1428] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
W20260927 23:06:02.536314 15692 tablet.cc:2369] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:06:02.578655 15826 tablet.cc:2369] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:06:02.639139 15549 tablet.cc:2369] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:06:02.695740 15851 raft_consensus.cc:482] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:06:02.695991 15851 raft_consensus.cc:504] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.697497 15851 leader_election.cc:288] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e653c2a6aa9d4c72a1bdda8a4a9f4c73 (127.3.240.195:37643), b1228ba1689c4f4d99359e2ba47259b0 (127.3.240.193:45875)
I20260927 23:06:02.707228 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.698032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:34983 (local address 127.3.240.195:37643)
0927 23:06:02.698214 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.698221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.699151 (+ 930us) server_negotiation.cc:407] Connection header received
0927 23:06:02.699331 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.699337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.699431 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.699522 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:02.700867 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.701977 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.704801 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.705966 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.706133 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.706192 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.706753 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.706879 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 23:06:02.707377 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.697763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45875 (local address 127.3.240.194:59423)
0927 23:06:02.698977 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:02.699016 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:02.699194 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:02.699552 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:02.699557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:02.699626 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:02.700695 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.700709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.702135 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.702141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.704642 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.704655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.706232 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.706239 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.706518 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.706587 (+ 69us) client_negotiation.cc:769] Sending connection context
0927 23:06:02.706704 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 23:06:02.706869 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":237,"spinlock_wait_cycles":6528,"thread_start_us":826,"threads_started":1}
I20260927 23:06:02.707475 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.697870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:37643 (local address 127.3.240.194:34983)
0927 23:06:02.699042 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:02.699057 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:02.699193 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:02.699597 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:02.699603 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:02.699630 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:02.700701 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.700709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.702135 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.702141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.704642 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:02.704655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.706243 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.706249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:02.706523 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.706587 (+ 64us) client_negotiation.cc:769] Sending connection context
0927 23:06:02.706704 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 23:06:02.706869 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":230,"spinlock_wait_cycles":22272,"thread_start_us":707,"threads_started":1}
I20260927 23:06:02.707681 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.698016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:59423 (local address 127.3.240.193:45875)
0927 23:06:02.698214 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.698221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.699151 (+ 930us) server_negotiation.cc:407] Connection header received
0927 23:06:02.699332 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.699337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.699431 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.699523 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:02.700867 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.701987 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.704801 (+ 2814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.705967 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.706124 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.706192 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.706753 (+ 561us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.706879 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:06:02.708676 15780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956c56e248544c5e806e45d73d082ccd" candidate_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" is_pre_election: true
I20260927 23:06:02.708676 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956c56e248544c5e806e45d73d082ccd" candidate_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1228ba1689c4f4d99359e2ba47259b0" is_pre_election: true
I20260927 23:06:02.709257 15503 raft_consensus.cc:2454] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd8f9f00fd34640b56a009dd9a78a12 in term 0.
I20260927 23:06:02.709257 15780 raft_consensus.cc:2454] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3bd8f9f00fd34640b56a009dd9a78a12 in term 0.
I20260927 23:06:02.710346 15579 leader_election.cc:302] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3bd8f9f00fd34640b56a009dd9a78a12, b1228ba1689c4f4d99359e2ba47259b0, e653c2a6aa9d4c72a1bdda8a4a9f4c73; no voters:
I20260927 23:06:02.710798 15851 raft_consensus.cc:2781] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:06:02.710944 15851 raft_consensus.cc:482] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:06:02.711061 15851 raft_consensus.cc:3037] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:02.713079 15851 raft_consensus.cc:504] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.713645 15851 leader_election.cc:288] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [CANDIDATE]: Term 1 election: Requested vote from peers e653c2a6aa9d4c72a1bdda8a4a9f4c73 (127.3.240.195:37643), b1228ba1689c4f4d99359e2ba47259b0 (127.3.240.193:45875)
I20260927 23:06:02.714252 15503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956c56e248544c5e806e45d73d082ccd" candidate_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1228ba1689c4f4d99359e2ba47259b0"
I20260927 23:06:02.714248 15780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956c56e248544c5e806e45d73d082ccd" candidate_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73"
I20260927 23:06:02.714452 15780 raft_consensus.cc:3037] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:02.714452 15503 raft_consensus.cc:3037] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:02.717461 15780 raft_consensus.cc:2454] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd8f9f00fd34640b56a009dd9a78a12 in term 1.
I20260927 23:06:02.717461 15503 raft_consensus.cc:2454] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3bd8f9f00fd34640b56a009dd9a78a12 in term 1.
I20260927 23:06:02.717856 15579 leader_election.cc:302] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bd8f9f00fd34640b56a009dd9a78a12, b1228ba1689c4f4d99359e2ba47259b0; no voters:
I20260927 23:06:02.718091 15851 raft_consensus.cc:2781] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:06:02.718497 15851 raft_consensus.cc:686] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 1 LEADER]: Becoming Leader. State: Replica: 3bd8f9f00fd34640b56a009dd9a78a12, State: Running, Role: LEADER
I20260927 23:06:02.718918 15851 consensus_queue.cc:237] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } }
I20260927 23:06:02.723423 15334 catalog_manager.cc:5495] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3bd8f9f00fd34640b56a009dd9a78a12 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3bd8f9f00fd34640b56a009dd9a78a12" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 } health_report { overall_health: UNKNOWN } } }
I20260927 23:06:02.773240 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.767536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55530 (local address 127.3.240.194:46089)
0927 23:06:02.767740 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.767745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.767764 (+ 19us) server_negotiation.cc:407] Connection header received
0927 23:06:02.767953 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.767957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.768028 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.768181 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:06:02.768544 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.769237 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.771179 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.771571 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.771723 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.772191 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:06:02.772663 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:06:02.772824 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.772886 (+ 62us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.772970 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:06:02.781272 15780 raft_consensus.cc:1261] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 1 FOLLOWER]: Refusing update from remote peer 3bd8f9f00fd34640b56a009dd9a78a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:06:02.781276 15503 raft_consensus.cc:1261] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 1 FOLLOWER]: Refusing update from remote peer 3bd8f9f00fd34640b56a009dd9a78a12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:06:02.782282 15855 consensus_queue.cc:1035] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1228ba1689c4f4d99359e2ba47259b0" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:06:02.782720 15851 consensus_queue.cc:1035] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e653c2a6aa9d4c72a1bdda8a4a9f4c73" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 37643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:06:02.812870 15860 mvcc.cc:205] Tried to move back new op lower bound from 7334094285942079488 to 7334094285700390912. Current Snapshot: MvccSnapshot[applied={T|T < 7334094285942079488}]
I20260927 23:06:02.817309 15334 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:06:02.817539 15334 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:06:02.820820 15334 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f7f292167394378ae972ceba6c98983: Sending DeleteTablet for 3 replicas of tablet 956c56e248544c5e806e45d73d082ccd
I20260927 23:06:02.821772 15626 tablet_service.cc:1514] Processing DeleteTablet for tablet 956c56e248544c5e806e45d73d082ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:06:02 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:55520
I20260927 23:06:02.821910 15760 tablet_service.cc:1514] Processing DeleteTablet for tablet 956c56e248544c5e806e45d73d082ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:06:02 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41018
I20260927 23:06:02.822142 15483 tablet_service.cc:1514] Processing DeleteTablet for tablet 956c56e248544c5e806e45d73d082ccd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:06:02 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36726
I20260927 23:06:02.824888 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 15415
I20260927 23:06:02.851678 15871 tablet_replica.cc:330] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: stopping tablet replica
I20260927 23:06:02.855219 15871 raft_consensus.cc:2229] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:06:02.855984 15871 raft_consensus.cc:2258] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:06:02.858777 15871 ts_tablet_manager.cc:1905] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:06:02.863651 15869 tablet_replica.cc:330] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: stopping tablet replica
I20260927 23:06:02.864135 15869 raft_consensus.cc:2229] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:06:02.864838 15869 raft_consensus.cc:2258] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:06:02.868863 15869 ts_tablet_manager.cc:1905] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:06:02.870354 15871 ts_tablet_manager.cc:1918] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:06:02.870513 15871 log.cc:1196] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/956c56e248544c5e806e45d73d082ccd
I20260927 23:06:02.870978 15871 ts_tablet_manager.cc:1939] T 956c56e248544c5e806e45d73d082ccd P b1228ba1689c4f4d99359e2ba47259b0: Deleting consensus metadata
Sep 27 23:06:02 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:06:02.872958 15870 tablet_replica.cc:330] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: stopping tablet replica
I20260927 23:06:02.873226 15318 catalog_manager.cc:4841] TS b1228ba1689c4f4d99359e2ba47259b0 (127.3.240.193:45875): tablet 956c56e248544c5e806e45d73d082ccd (table test-table [id=afbe214edeea4c60818ca8930af0ab85]) successfully deleted
I20260927 23:06:02.873345 15870 raft_consensus.cc:2229] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:06:02.873757 15870 raft_consensus.cc:2258] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:06:02.875416 15870 ts_tablet_manager.cc:1905] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:06:02.882880 15870 ts_tablet_manager.cc:1918] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:06:02.882957 15869 ts_tablet_manager.cc:1918] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:06:02.883134 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.826652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.3.240.193:45875)
0927 23:06:02.826865 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.826874 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.850854 (+ 23980us) server_negotiation.cc:407] Connection header received
0927 23:06:02.852197 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.852207 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.852312 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.852401 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:02.859062 (+ 6661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.860464 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.865052 (+ 4588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.865737 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.865828 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.875483 (+ 9655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:02.875498 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:02.875515 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:02.875558 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:02.878908 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:02.879854 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:02.879864 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:02.879868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:02.880015 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:02.880480 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:02.880485 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:02.880488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:02.881042 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:02.881742 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:02.882380 (+ 638us) server_negotiation.cc:299] Negotiation successful
0927 23:06:02.882657 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:06:02.883150 15870 log.cc:1196] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/956c56e248544c5e806e45d73d082ccd
I20260927 23:06:02.883692 15870 ts_tablet_manager.cc:1939] T 956c56e248544c5e806e45d73d082ccd P e653c2a6aa9d4c72a1bdda8a4a9f4c73: Deleting consensus metadata
I20260927 23:06:02.884099 15869 log.cc:1196] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/956c56e248544c5e806e45d73d082ccd
I20260927 23:06:02.884526 15543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:02.884635 15869 ts_tablet_manager.cc:1939] T 956c56e248544c5e806e45d73d082ccd P 3bd8f9f00fd34640b56a009dd9a78a12: Deleting consensus metadata
I20260927 23:06:02.886302 15318 catalog_manager.cc:4841] TS 3bd8f9f00fd34640b56a009dd9a78a12 (127.3.240.194:46089): tablet 956c56e248544c5e806e45d73d082ccd (table test-table [id=afbe214edeea4c60818ca8930af0ab85]) successfully deleted
I20260927 23:06:02.888700 15321 catalog_manager.cc:4841] TS e653c2a6aa9d4c72a1bdda8a4a9f4c73 (127.3.240.195:37643): tablet 956c56e248544c5e806e45d73d082ccd (table test-table [id=afbe214edeea4c60818ca8930af0ab85]) successfully deleted
I20260927 23:06:02.970872 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 15415
I20260927 23:06:02.988986 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 15554
Sep 27 23:06:02 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:06:03.004698 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:02.990073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.3.240.194:46089)
0927 23:06:02.990245 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:02.990250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:02.990264 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:06:02.990314 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:02.990318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:02.990375 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:02.990481 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:02.991801 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.992518 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.993720 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:02.994148 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:02.994280 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:02.998559 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:02.998571 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:02.998586 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:02.998622 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:03.001616 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:03.002310 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:03.002318 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:03.002320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:03.002467 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:03.002860 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:03.002865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:03.002869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:03.003512 (+ 643us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:03.003865 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:03.004131 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 23:06:03.004343 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 23:06:03.005523 15686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:03.120926 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 15554
I20260927 23:06:03.138727 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 15697
Sep 27 23:06:03 dist-test-slave-lmfp krb5kdc[14858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:06:03.153780 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:03.139622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41022 (local address 127.3.240.195:37643)
0927 23:06:03.139829 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:03.139838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:03.139924 (+ 86us) server_negotiation.cc:407] Connection header received
0927 23:06:03.140168 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:03.140173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:03.140244 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:03.140363 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:03.141772 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:03.142485 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:03.143921 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:03.144309 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:03.144442 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:03.148573 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:03.148586 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:03.148602 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:03.148633 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:03.150978 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:03.151706 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:03.151722 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:03.151725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:03.151840 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:03.152232 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:03.152243 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:03.152246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:03.152595 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:03.152889 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:03.153261 (+ 372us) server_negotiation.cc:299] Negotiation successful
0927 23:06:03.153465 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:06:03.154373 15820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:03.240612 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 15697
I20260927 23:06:03.256580 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 15302
I20260927 23:06:03.257856 15364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:03.366551 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 15302
I20260927 23:06:03.390086 4035 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 23:06:03.724624 4035 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 23:06:03.724838 4035 mini_ranger.cc:66] Stopping Ranger...
I20260927 23:06:04.069456 4035 mini_ranger.cc:68] Stopped Ranger
2026-09-27 23:06:04.069 UTC [14888] LOG: received smart shutdown request
2026-09-27 23:06:04.072 UTC [14888] LOG: background worker "logical replication launcher" (PID 14897) exited with exit code 1
2026-09-27 23:06:04.073 UTC [14892] LOG: shutting down
2026-09-27 23:06:04.223 UTC [14888] LOG: database system is shut down
2026-09-27T23:06:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46846 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:06:04 dist-test-slave-lmfp krb5kdc[15885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:06:04 dist-test-slave-lmfp krb5kdc[15885](info): set up 2 sockets
Sep 27 23:06:04 dist-test-slave-lmfp krb5kdc[15885](info): commencing operation
krb5kdc: starting...
W20260927 23:06:05.435384 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:06:05 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:06:05Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/rangeradmin_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/rangeradmin_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/rangerlookup_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/rangerlookup_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
I20260927 23:06:05.735735 4035 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task9OvN8C/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/postgres -l logfile start
2026-09-27 23:06:08.107 UTC [15915] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-27 23:06:08.107 UTC [15915] LOG: listening on IPv4 address "127.3.240.212", port 33583
2026-09-27 23:06:08.111 UTC [15915] LOG: listening on Unix socket "/tmp/.s.PGSQL.33583"
2026-09-27 23:06:08.132 UTC [15918] LOG: database system was shut down at 2026-09-27 23:06:06 UTC
2026-09-27 23:06:08.136 UTC [15915] LOG: database system is ready to accept connections
I20260927 23:06:09.059156 4035 mini_postgres.cc:96] Postgres bound to 33583
2026-09-27 23:06:09.076 UTC [15926] FATAL: database "slave" does not exist
127.3.240.212:33583 - accepting connections
I20260927 23:06:09.077598 4035 mini_ranger.cc:161] Starting Ranger...
I20260927 23:06:09.102892 4035 mini_ranger.cc:84] Created miniranger Postgres user
I20260927 23:06:09.555857 4035 mini_ranger.cc:87] Created ranger Postgres database
I20260927 23:06:09.555975 4035 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-admin
2026-09-27 23:06:09,851 [I] DB FLAVOR :POSTGRES
2026-09-27 23:06:09,852 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 23:06:09,852 [I] Checking connection..
2026-09-27 23:06:09,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-27 23:06:10,080 [I] Checking connection passed.
2026-09-27 23:06:10,080 [I] --------- Verifying version history table ---------
2026-09-27 23:06:10,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 23:06:10,315 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 23:06:10,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 23:06:10,545 [I] Table x_db_version_h does not exist in database ranger
2026-09-27 23:06:10,546 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-27 23:06:10,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-27 23:06:10.770 UTC [15990] WARNING: there is no transaction in progress
2026-09-27 23:06:10,775 [I] create_dbversion_catalog.sql file imported successfully
2026-09-27 23:06:10,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-27 23:06:11,027 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-27 23:06:11,027 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 23:06:11,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-27 23:06:11,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-27 23:06:11,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lmfp.c.gcp-upstream.internal','N') ;"
2026-09-27 23:06:11,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 23:06:11,956 [I] Table x_portal_user does not exist in database ranger
2026-09-27 23:06:11,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 23:06:12,196 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-27 23:06:12,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 23:06:12,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 23:06:12,658 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-27 23:06:12,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-27 23:06:13.543 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.559 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.574 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.589 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.607 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.625 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.676 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.682 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.695 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.704 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.714 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.726 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.735 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.741 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.747 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:13.753 UTC [16098] WARNING: there is no transaction in progress
2026-09-27 23:06:14,314 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-27 23:06:14,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-27 23:06:14,534 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-27 23:06:14,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-27 23:06:14,770 [I] Table x_portal_user already exists in database 'ranger'
2026-09-27 23:06:14,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-27 23:06:15,011 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-27 23:06:15,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-27 23:06:15,255 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-27 23:06:15,494 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lmfp.c.gcp-upstream.internal';"
2026-09-27 23:06:15,717 [I] CORE_DB_SCHEMA import status has been updated
2026-09-27 23:06:15,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-27 23:06:15,935 [I] DB_PATCHES have already been applied
I20260927 23:06:15.941625 4035 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-admin:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task9OvN8C/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260927 23:06:15.941708 4035 mini_ranger.cc:193] Using host: 127.3.240.212
I20260927 23:06:15.955433 4035 mini_ranger.cc:227] Ranger admin URL: http://127.3.240.212:60887
Sep 27, 2026 11:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-admin/ews/webapp
Sep 27, 2026 11:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 27, 2026 11:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 27, 2026 11:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-admin/ews/webapp].
Sep 27, 2026 11:06:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260927 23:06:17.050457 4035 mini_ranger.cc:160] Time spent starting Ranger: real 7.973s user 0.002s sys 0.033s
Sep 27, 2026 11:06:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 27, 2026 11:06:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 27, 2026 11:06:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 27, 2026 11:06:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 27, 2026 11:06:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260927 23:06:36.695467 4035 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/rangerkms_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/rangerkms_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.240.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.240.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 27 23:06:36 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550396, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260927 23:06:36.864856 4035 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260927 23:06:36.887090 4035 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260927 23:06:37.706830 4035 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260927 23:06:37.706959 4035 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms
I20260927 23:06:37.707037 4035 mini_ranger_kms.cc:227] Using postgres at 127.3.240.212:33583
2026-09-27 23:06:38,041 [I] DB FLAVOR :POSTGRES
2026-09-27 23:06:38,041 [I] --------- Verifying Ranger DB connection ---------
2026-09-27 23:06:38,042 [I] Checking connection
2026-09-27 23:06:38,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-27 23:06:38,263 [I] connection success
2026-09-27 23:06:38,263 [I] --------- Verifying Ranger DB tables ---------
2026-09-27 23:06:38,263 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-27 23:06:38,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-27 23:06:38,484 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-27 23:06:38,484 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-27 23:06:38,485 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-27 23:06:38,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:33583/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-27 23:06:38,745 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260927 23:06:39.171087 4035 mini_ranger_kms.cc:310] Created kms service
I20260927 23:06:39.330564 4035 mini_ranger_kms.cc:326] Created kudu user
I20260927 23:06:39.374589 4035 mini_ranger_kms.cc:343] Created rangerkms user
Sep 27 23:06:39 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550396, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:39.767591 4035 mini_ranger_kms.cc:384] Added ranger policy
I20260927 23:06:39.767925 4035 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms:/tmp/dist-test-task9OvN8C/build/asan/bin/postgresql.jar:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task9OvN8C/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260927 23:06:39.767993 4035 mini_ranger_kms.cc:240] Using host: 127.3.240.212
I20260927 23:06:39.784004 4035 mini_ranger_kms.cc:276] Ranger KMS PID: 16278
I20260927 23:06:39.784121 4035 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.240.212:54791
Sep 27, 2026 11:06:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 27, 2026 11:06:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 27, 2026 11:06:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 27, 2026 11:06:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 27, 2026 11:06:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-kms/ews/webapp].
Sep 27, 2026 11:06:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 27 23:06:41 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790550401, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:41.990985 4035 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.126s user 0.078s sys 0.274s
I20260927 23:06:41.991271 4035 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260927 23:06:41.991470 4035 mini_ranger_kms.cc:401] 127.3.240.212:54791/kms/v1/keys
Sep 27 23:06:42 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550401, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Sep 27 23:06:42 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550401, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Sep 27 23:06:44 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:06:44.983243 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35221
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:35221
--ranger_config_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:54791/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:06:45.295599 16329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:45.295871 16329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:45.296032 16329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:45.303031 16329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:06:45.303146 16329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:45.303220 16329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:45.303323 16329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:06:45.303409 16329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:06:45.313143 16329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.240.212:54791/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:35221
--ranger_config_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:45.314332 16329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:45.315657 16329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:45.326900 16337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:45.326936 16335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:45.326956 16334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:45.327630 16329 server_base.cc:1034] running on GCE node
I20260927 23:06:45.328258 16329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:45.329454 16329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:45.330662 16329 hybrid_clock.cc:648] HybridClock initialized: now 1790550405330618 us; error 38 us; skew 500 ppm
I20260927 23:06:45.331063 16329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:45 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550405, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:45.333889 16329 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:06:45.335237 16329 webserver.cc:460] Webserver started at http://127.3.240.254:43721/ using document root <none> and password file <none>
I20260927 23:06:45.335857 16329 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:45.335988 16329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:45.336277 16329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 23:06:45 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550405, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:45.399199 16329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "f8aec173c4764849b22d1db47b480299"
format_stamp: "Formatted at 2026-09-27 23:06:45 on dist-test-slave-lmfp"
server_key: "c9ea1bb8c9dcdf43420406201ac452bd"
server_key_iv: "658a4c9f39cc63afc24e8433fa49ecb9"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:45.400024 16329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "f8aec173c4764849b22d1db47b480299"
format_stamp: "Formatted at 2026-09-27 23:06:45 on dist-test-slave-lmfp"
server_key: "c9ea1bb8c9dcdf43420406201ac452bd"
server_key_iv: "658a4c9f39cc63afc24e8433fa49ecb9"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:45.405289 16329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260927 23:06:45.423913 16344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:45.425446 16329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:06:45.425616 16329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "f8aec173c4764849b22d1db47b480299"
format_stamp: "Formatted at 2026-09-27 23:06:45 on dist-test-slave-lmfp"
server_key: "c9ea1bb8c9dcdf43420406201ac452bd"
server_key_iv: "658a4c9f39cc63afc24e8433fa49ecb9"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:45.425819 16329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:45.443941 16329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:45.448235 16329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:45.448559 16329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:45.467545 16329 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:35221
I20260927 23:06:45.467597 16395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:35221 every 8 connection(s)
I20260927 23:06:45.468871 16329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:06:45.472579 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 16329
I20260927 23:06:45.472823 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:06:45.473308 16396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:45.473486 4035 external_mini_cluster.cc:1428] Setting key e3c03192e3f6f569682e2c0a30ee7897
I20260927 23:06:45.486301 16396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Bootstrap starting.
I20260927 23:06:45.489987 16396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:45.491220 16396 log.cc:824] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Log is configured to *not* fsync() on all Append() calls
Sep 27 23:06:45 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:45.494521 16396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: No bootstrap required, opened a new log
I20260927 23:06:45.500443 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:45.475521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58166 (local address 127.3.240.254:35221)
0927 23:06:45.480255 (+ 4734us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:45.480266 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:45.480303 (+ 37us) server_negotiation.cc:407] Connection header received
0927 23:06:45.481789 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:45.481811 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:45.482196 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:45.482498 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:45.484032 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:45.485243 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:45.487072 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:45.487760 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:45.487886 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:45.492716 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:45.492726 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:45.492743 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:45.492780 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:45.496321 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:45.496978 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:45.496986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:45.496994 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:45.497180 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:45.497510 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:45.497514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:45.497518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:45.498151 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:45.498436 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:45.498738 (+ 302us) server_negotiation.cc:299] Negotiation successful
0927 23:06:45.499130 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4521,"thread_start_us":4262,"threads_started":1}
I20260927 23:06:45.503665 16396 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } }
I20260927 23:06:45.503988 16396 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:45.504060 16396 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aec173c4764849b22d1db47b480299, State: Initialized, Role: FOLLOWER
I20260927 23:06:45.504508 16396 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } }
I20260927 23:06:45.504702 16396 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:06:45.504796 16396 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:06:45.504920 16396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:45.506913 16396 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } }
I20260927 23:06:45.507403 16396 leader_election.cc:302] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8aec173c4764849b22d1db47b480299; no voters:
I20260927 23:06:45.507890 16396 leader_election.cc:288] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:06:45.508045 16401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:06:45.508703 16401 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 1 LEADER]: Becoming Leader. State: Replica: f8aec173c4764849b22d1db47b480299, State: Running, Role: LEADER
I20260927 23:06:45.509167 16401 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } }
I20260927 23:06:45.509811 16396 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:06:45.512563 16402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8aec173c4764849b22d1db47b480299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } } }
I20260927 23:06:45.512588 16403 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8aec173c4764849b22d1db47b480299. Latest consensus state: current_term: 1 leader_uuid: "f8aec173c4764849b22d1db47b480299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } } }
I20260927 23:06:45.513362 16402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:45.513453 16403 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:45.514284 16408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:06:45.519239 16408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:06:45.519937 16396 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260927 23:06:45.536049 16408 catalog_manager.cc:1347] Generated new cluster ID: 500a6f944fed4cdfb72897be775d8387
I20260927 23:06:45.536183 16408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:06:45.567006 16408 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:06:45.568883 16408 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:06:45.581414 16408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Generated new TSK 0
I20260927 23:06:45.582438 16408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:06:46.243420 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:54791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35221
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:06:46 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790550406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260927 23:06:46.625969 16441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:46.626363 16441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:46.626561 16441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:46.634065 16441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:46.634189 16441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:46.634363 16441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:06:46.646036 16441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.240.212:54791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:46.647410 16441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:46.648691 16441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:46.665621 16451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:46.666523 16452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:46.674592 16454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:46.675387 16441 server_base.cc:1034] running on GCE node
I20260927 23:06:46.676055 16441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:46.677115 16441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:46.678330 16441 hybrid_clock.cc:648] HybridClock initialized: now 1790550406678284 us; error 41 us; skew 500 ppm
I20260927 23:06:46.678689 16441 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:46 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:46.682780 16441 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:06:46.684792 16441 webserver.cc:460] Webserver started at http://127.3.240.193:41527/ using document root <none> and password file <none>
I20260927 23:06:46.685735 16441 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:46.685870 16441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:46.686195 16441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 23:06:46 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:46.710632 16441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "0ab24245470743cd86314e4d2ff43d30"
format_stamp: "Formatted at 2026-09-27 23:06:46 on dist-test-slave-lmfp"
server_key: "0b7d627106562e600cbda138480495cb"
server_key_iv: "258e4a0cde58545004fa764953da195a"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:46.713066 16441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "0ab24245470743cd86314e4d2ff43d30"
format_stamp: "Formatted at 2026-09-27 23:06:46 on dist-test-slave-lmfp"
server_key: "0b7d627106562e600cbda138480495cb"
server_key_iv: "258e4a0cde58545004fa764953da195a"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:46.720440 16441 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260927 23:06:46.755700 16461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:46.757606 16441 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.015s sys 0.000s
I20260927 23:06:46.757858 16441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "0ab24245470743cd86314e4d2ff43d30"
format_stamp: "Formatted at 2026-09-27 23:06:46 on dist-test-slave-lmfp"
server_key: "0b7d627106562e600cbda138480495cb"
server_key_iv: "258e4a0cde58545004fa764953da195a"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:46.758206 16441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:46.783520 16441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:46.788694 16441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:46.789052 16441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:46.795380 16441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:46.800870 16441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:46.801017 16441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 23:06:46.801189 16441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:46.801294 16441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:46.870014 16573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:41181 every 8 connection(s)
I20260927 23:06:46.870407 16441 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:41181
I20260927 23:06:46.872131 16441 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:06:46.882748 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 16441
I20260927 23:06:46.883059 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:06:46.884028 4035 external_mini_cluster.cc:1428] Setting key 2157485b2c7c044a26978b12622ebfe1
Sep 27 23:06:46 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:46.934648 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:46.877348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:52467 (local address 127.3.240.254:35221)
0927 23:06:46.915726 (+ 38378us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:46.915747 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:46.915788 (+ 41us) server_negotiation.cc:407] Connection header received
0927 23:06:46.915899 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:46.915908 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:46.916079 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:46.916248 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:46.918566 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:46.919802 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:46.922586 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:46.923311 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:46.924362 (+ 1051us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:46.927938 (+ 3576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:46.927944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:46.927948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:46.927991 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:46.931614 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:46.932358 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:46.932363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:46.932369 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:46.932486 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:46.932941 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:46.932946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:46.932950 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:46.933215 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:46.933368 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:46.934067 (+ 699us) server_negotiation.cc:299] Negotiation successful
0927 23:06:46.934272 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34152,"thread_start_us":34759,"threads_started":1}
I20260927 23:06:46.935923 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:46.876516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35221 (local address 127.3.240.193:52467)
0927 23:06:46.877321 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:46.877393 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:46.881854 (+ 4461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:46.916649 (+ 34795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:46.916664 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:46.917369 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:46.918300 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:46.918330 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:46.920001 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:46.920010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:46.922373 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:46.922392 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:46.923583 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:46.923590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:46.923879 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:46.924709 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:46.924747 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:46.927605 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:46.931766 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:46.931776 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:46.931793 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:46.932222 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:46.932604 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:46.932608 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:46.932612 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:46.932821 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:46.933346 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:46.933354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:46.933692 (+ 338us) client_negotiation.cc:769] Sending connection context
0927 23:06:46.933971 (+ 279us) client_negotiation.cc:240] Negotiation successful
0927 23:06:46.934485 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":822,"threads_started":1}
I20260927 23:06:46.937821 16574 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35221
I20260927 23:06:46.938155 16574 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:46.939141 16574 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
I20260927 23:06:46.941454 16361 ts_manager.cc:194] Registered new tserver with Master: 0ab24245470743cd86314e4d2ff43d30 (127.3.240.193:41181)
I20260927 23:06:46.942965 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:52467
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:06:46.985329 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:54791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35221
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:06:46 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Sep 27 23:06:47 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550406, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
W20260927 23:06:47.437613 16580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:47.438134 16580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:47.438331 16580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:47.448798 16580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:47.448975 16580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:47.449165 16580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:06:47.464231 16580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.240.212:54791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:47.465941 16580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:47.467561 16580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:47.494392 16588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:47.494546 16587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:47.504374 16590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:47.504604 16580 server_base.cc:1034] running on GCE node
I20260927 23:06:47.505426 16580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:47.516498 16580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:47.517760 16580 hybrid_clock.cc:648] HybridClock initialized: now 1790550407517693 us; error 59 us; skew 500 ppm
I20260927 23:06:47.518199 16580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:47 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:47.522265 16580 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:06:47.530679 16580 webserver.cc:460] Webserver started at http://127.3.240.194:37201/ using document root <none> and password file <none>
I20260927 23:06:47.531636 16580 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:47.531764 16580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:47.532110 16580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 23:06:47 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:47.549751 16580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "788bb0e3dfb442a8954bd403fd567112"
format_stamp: "Formatted at 2026-09-27 23:06:47 on dist-test-slave-lmfp"
server_key: "9910d26246dd2ecb2c82b290d205804d"
server_key_iv: "cf90ed10f0f12cbff1bcd19d8e6d3ebd"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:47.550550 16580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "788bb0e3dfb442a8954bd403fd567112"
format_stamp: "Formatted at 2026-09-27 23:06:47 on dist-test-slave-lmfp"
server_key: "9910d26246dd2ecb2c82b290d205804d"
server_key_iv: "cf90ed10f0f12cbff1bcd19d8e6d3ebd"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:47.556278 16580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260927 23:06:47.576526 16598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:47.578342 16580 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260927 23:06:47.578528 16580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "788bb0e3dfb442a8954bd403fd567112"
format_stamp: "Formatted at 2026-09-27 23:06:47 on dist-test-slave-lmfp"
server_key: "9910d26246dd2ecb2c82b290d205804d"
server_key_iv: "cf90ed10f0f12cbff1bcd19d8e6d3ebd"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:47.578785 16580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:47.617024 16580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:47.620560 16580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:47.620850 16580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:47.621759 16580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:47.623080 16580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:47.623148 16580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:47.623320 16580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:47.623378 16580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:47.655923 16580 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:46155
I20260927 23:06:47.655961 16716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:46155 every 8 connection(s)
I20260927 23:06:47.657183 16580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:06:47.657612 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 16580
I20260927 23:06:47.657824 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:06:47.658538 4035 external_mini_cluster.cc:1428] Setting key b33af8486cf704e106a898baf82faa67
Sep 27 23:06:47 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550407, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:47.679225 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:47.659972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:49407 (local address 127.3.240.254:35221)
0927 23:06:47.660530 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:47.660537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:47.661676 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 23:06:47.664544 (+ 2868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:47.664549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:47.664612 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:47.664709 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:47.666754 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:47.667645 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:47.669654 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:47.670048 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:47.670181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:47.673493 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:47.673499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:47.673502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:47.673532 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:47.676218 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:47.676960 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:47.676970 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:47.676973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:47.677096 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:47.677508 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:47.677516 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:47.677520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:47.677790 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:47.677943 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:47.678586 (+ 643us) server_negotiation.cc:299] Negotiation successful
0927 23:06:47.678812 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":293,"threads_started":1}
I20260927 23:06:47.680263 16720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:47.660648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35221 (local address 127.3.240.194:49407)
0927 23:06:47.661496 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:47.661556 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:47.664239 (+ 2683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:47.665003 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:47.665019 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:47.665658 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:47.666543 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:47.666572 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:47.667820 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:47.667828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:47.669501 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:47.669512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:47.670275 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:47.670282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:47.670512 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:47.671273 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:47.671295 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:47.673272 (+ 1977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:47.676376 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:47.676386 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:47.676404 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:47.676823 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:47.677213 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:47.677219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:47.677222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:47.677400 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:47.677915 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:47.677922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:47.678232 (+ 310us) client_negotiation.cc:769] Sending connection context
0927 23:06:47.678493 (+ 261us) client_negotiation.cc:240] Negotiation successful
0927 23:06:47.679024 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":323,"threads_started":1}
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260927 23:06:47.681819 16717 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35221
I20260927 23:06:47.682118 16717 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:47.682770 16717 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
I20260927 23:06:47.684381 16361 ts_manager.cc:194] Registered new tserver with Master: 788bb0e3dfb442a8954bd403fd567112 (127.3.240.194:46155)
I20260927 23:06:47.685264 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:49407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:06:47.710038 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:54791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35221
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:06:47.946358 16574 heartbeater.cc:503] Master 127.3.240.254:35221 was elected leader, sending a full tablet report...
W20260927 23:06:48.019404 16723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:48.019690 16723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:48.019845 16723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:48.026780 16723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:48.026916 16723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:48.027107 16723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:06:48.036381 16723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.240.212:54791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:48.037580 16723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:48.038862 16723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:48.051617 16728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:48.051569 16729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:48.051872 16731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:48.052464 16723 server_base.cc:1034] running on GCE node
I20260927 23:06:48.052951 16723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:48.053745 16723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:48.054934 16723 hybrid_clock.cc:648] HybridClock initialized: now 1790550408054899 us; error 34 us; skew 500 ppm
I20260927 23:06:48.055275 16723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:48 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:48.058094 16723 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:06:48.059547 16723 webserver.cc:460] Webserver started at http://127.3.240.195:36165/ using document root <none> and password file <none>
I20260927 23:06:48.060153 16723 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:48.060283 16723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:48.060554 16723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 27 23:06:48 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:48.076910 16723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "edb4af071ce64df7b5fb92fd46a71d5c"
format_stamp: "Formatted at 2026-09-27 23:06:48 on dist-test-slave-lmfp"
server_key: "8982c6a6bc6e76d0e3ea8da6a462ca6c"
server_key_iv: "bf86335ff3c6673f6c2741032418ad0e"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:48.077772 16723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "edb4af071ce64df7b5fb92fd46a71d5c"
format_stamp: "Formatted at 2026-09-27 23:06:48 on dist-test-slave-lmfp"
server_key: "8982c6a6bc6e76d0e3ea8da6a462ca6c"
server_key_iv: "bf86335ff3c6673f6c2741032418ad0e"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:48.083038 16723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260927 23:06:48.098277 16738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:48.099792 16723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:06:48.099920 16723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "edb4af071ce64df7b5fb92fd46a71d5c"
format_stamp: "Formatted at 2026-09-27 23:06:48 on dist-test-slave-lmfp"
server_key: "8982c6a6bc6e76d0e3ea8da6a462ca6c"
server_key_iv: "bf86335ff3c6673f6c2741032418ad0e"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:48.100080 16723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:48.125494 16723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:48.128925 16723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:48.129148 16723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:48.130028 16723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:48.131335 16723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:48.131410 16723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:48.131487 16723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:48.131531 16723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:48.163372 16723 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42299
I20260927 23:06:48.163419 16850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42299 every 8 connection(s)
I20260927 23:06:48.164832 16723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:06:48.168936 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 16723
I20260927 23:06:48.169134 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:06:48.169624 4035 external_mini_cluster.cc:1428] Setting key a3a8ec8c96445cfac9c0a78c8e48e046
Sep 27 23:06:48 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550408, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:48.183548 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:48.167404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:42743 (local address 127.3.240.254:35221)
0927 23:06:48.167576 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:48.167581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:48.168730 (+ 1149us) server_negotiation.cc:407] Connection header received
0927 23:06:48.170643 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:48.170648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:48.170777 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:48.170883 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:48.172252 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.172984 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.174468 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.174863 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.174993 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:48.177924 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:48.177929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:48.177932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:48.177961 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:48.180379 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:48.181225 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:48.181232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:48.181275 (+ 43us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:48.181432 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:48.181871 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:48.181876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:48.181879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:48.182153 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:48.182325 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:48.183018 (+ 693us) server_negotiation.cc:299] Negotiation successful
0927 23:06:48.183227 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:06:48.184588 16853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:48.167926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35221 (local address 127.3.240.195:42743)
0927 23:06:48.168577 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:48.168612 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:48.170346 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:48.171147 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:48.171155 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:48.171543 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:48.172081 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:48.172100 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.173138 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.173148 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:48.174321 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:48.174331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.175089 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.175096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:48.175313 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:48.175876 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:48.175895 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:48.177711 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:48.180520 (+ 2809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:48.180528 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:48.180542 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:48.181111 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:48.181568 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:48.181574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:48.181578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:48.181749 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:48.182308 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:48.182316 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:48.182649 (+ 333us) client_negotiation.cc:769] Sending connection context
0927 23:06:48.182898 (+ 249us) client_negotiation.cc:240] Negotiation successful
0927 23:06:48.183380 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":260,"threads_started":1}
I20260927 23:06:48.185796 16851 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35221
I20260927 23:06:48.186080 16851 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:48.186723 16851 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
I20260927 23:06:48.188010 16361 ts_manager.cc:194] Registered new tserver with Master: edb4af071ce64df7b5fb92fd46a71d5c (127.3.240.195:42299)
I20260927 23:06:48.188803 16361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:42743
I20260927 23:06:48.196674 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:06:48.198323 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 16441
Sep 27 23:06:48 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:06:48.215116 16855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:48.199243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.3.240.193:41181)
0927 23:06:48.199857 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:48.199864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:48.199888 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:06:48.199941 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:48.199947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:48.200140 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:48.200281 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:48.201688 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.202745 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.204426 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.205118 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.205230 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:48.209429 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:48.209441 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:48.209456 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:48.209491 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:48.212549 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:48.213226 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:48.213232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:48.213235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:48.213370 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:48.213698 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:48.213701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:48.213703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:48.214012 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:48.214232 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:48.214463 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 23:06:48.214654 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":339,"threads_started":1}
I20260927 23:06:48.216459 16569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:48.299217 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 16441
I20260927 23:06:48.315541 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 16580
Sep 27 23:06:48 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:06:48.331676 16857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:48.316553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.3.240.194:46155)
0927 23:06:48.317090 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:48.317097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:48.317122 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:06:48.317178 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:48.317184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:48.317426 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:48.317549 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:48.318639 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.319715 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.321103 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.321684 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.321794 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:48.325979 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:48.325989 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:48.326003 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:48.326032 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:48.328806 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:48.329547 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:48.329554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:48.329556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:48.329678 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:48.330071 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:48.330074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:48.330076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:48.330488 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:48.330712 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:48.330951 (+ 239us) server_negotiation.cc:299] Negotiation successful
0927 23:06:48.331284 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":280,"threads_started":1}
I20260927 23:06:48.332901 16712 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:48.412542 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 16580
I20260927 23:06:48.428298 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 16723
Sep 27 23:06:48 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:06:48.443333 16859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:48.429205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.3.240.195:42299)
0927 23:06:48.429741 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:48.429747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:48.429774 (+ 27us) server_negotiation.cc:407] Connection header received
0927 23:06:48.429831 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:48.429836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:48.430045 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:48.430164 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:48.431535 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.432525 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.433645 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:48.434153 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:48.434243 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:48.438311 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:48.438323 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:48.438339 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:48.438370 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:48.440990 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:48.441634 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:48.441641 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:48.441643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:48.441756 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:48.442084 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:48.442087 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:48.442089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:48.442387 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:48.442617 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:48.442785 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 23:06:48.442974 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":251,"threads_started":1}
I20260927 23:06:48.444536 16846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:48.524642 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 16723
I20260927 23:06:48.539932 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 16329
I20260927 23:06:48.541134 16391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:48.644517 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 16329
I20260927 23:06:48.665967 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35221
--webserver_interface=127.3.240.254
--webserver_port=43721
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:35221
--ranger_config_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:54791/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:54791/kms with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:06:48.970377 16862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:48.970659 16862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:48.970829 16862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:48.978018 16862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:06:48.978121 16862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:48.978248 16862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:48.978324 16862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:06:48.978396 16862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:06:48.988463 16862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:54791/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:35221
--ranger_config_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=43721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:48.989743 16862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:48.991052 16862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:49.002502 16870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:49.002537 16868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:49.002617 16867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:49.003237 16862 server_base.cc:1034] running on GCE node
I20260927 23:06:49.003931 16862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:49.005157 16862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:49.006369 16862 hybrid_clock.cc:648] HybridClock initialized: now 1790550409006323 us; error 33 us; skew 500 ppm
I20260927 23:06:49.006718 16862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:49 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:49.009696 16862 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:06:49.010908 16862 webserver.cc:460] Webserver started at http://127.3.240.254:43721/ using document root <none> and password file <none>
I20260927 23:06:49.011571 16862 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:49.011718 16862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 23:06:49 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:49.076377 16862 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:06:49.079222 16878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:49.080494 16862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 23:06:49.080667 16862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "f8aec173c4764849b22d1db47b480299"
format_stamp: "Formatted at 2026-09-27 23:06:45 on dist-test-slave-lmfp"
server_key: "c9ea1bb8c9dcdf43420406201ac452bd"
server_key_iv: "658a4c9f39cc63afc24e8433fa49ecb9"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:49.081584 16862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:49.112948 16862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:49.116999 16862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:49.117271 16862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:49.135291 16862 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:35221
I20260927 23:06:49.135313 16929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:35221 every 8 connection(s)
I20260927 23:06:49.136703 16862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:06:49.137530 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 16862
I20260927 23:06:49.138549 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:54791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:41181
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:35221
--webserver_port=41527
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:54791/kms with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:06:49.144376 16930 sys_catalog.cc:263] Verifying existing consensus state
I20260927 23:06:49.147609 16930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Bootstrap starting.
I20260927 23:06:49.166982 16930 log.cc:824] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:49.175184 16930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:06:49.175660 16930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Bootstrap complete.
I20260927 23:06:49.184176 16930 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } }
I20260927 23:06:49.185303 16930 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aec173c4764849b22d1db47b480299, State: Initialized, Role: FOLLOWER
I20260927 23:06:49.186000 16930 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } }
I20260927 23:06:49.186241 16930 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:06:49.186367 16930 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:06:49.186546 16930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:06:49.190100 16930 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } }
I20260927 23:06:49.190579 16930 leader_election.cc:302] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8aec173c4764849b22d1db47b480299; no voters:
I20260927 23:06:49.191200 16930 leader_election.cc:288] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 23:06:49.191356 16934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:06:49.192430 16934 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [term 2 LEADER]: Becoming Leader. State: Replica: f8aec173c4764849b22d1db47b480299, State: Running, Role: LEADER
I20260927 23:06:49.193027 16934 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } }
I20260927 23:06:49.193336 16930 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:06:49.195492 16935 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f8aec173c4764849b22d1db47b480299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } } }
I20260927 23:06:49.195905 16935 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:49.198931 16939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:06:49.198900 16936 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8aec173c4764849b22d1db47b480299. Latest consensus state: current_term: 2 leader_uuid: "f8aec173c4764849b22d1db47b480299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aec173c4764849b22d1db47b480299" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35221 } } }
I20260927 23:06:49.199240 16936 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299 [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:49.205636 16939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:06:49.207206 16939 catalog_manager.cc:1259] Loaded cluster ID: 500a6f944fed4cdfb72897be775d8387
I20260927 23:06:49.207355 16939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:06:49.211184 16939 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:06:49.212579 16939 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Loaded TSK: 0
I20260927 23:06:49.214208 16939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:06:49.220937 16930 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260927 23:06:49.565881 16932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:49.566347 16932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:49.566537 16932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:49.578787 16932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:49.578959 16932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:49.579226 16932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:06:49.594765 16932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:54791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:41181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=41527
--tserver_master_addrs=127.3.240.254:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:49.596520 16932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:49.598198 16932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:49.643110 16967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:49.644768 16968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:49.649911 16932 server_base.cc:1034] running on GCE node
W20260927 23:06:49.650640 16970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:49.651530 16932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:49.653417 16932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:49.654714 16932 hybrid_clock.cc:648] HybridClock initialized: now 1790550409654609 us; error 97 us; skew 500 ppm
I20260927 23:06:49.655369 16932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:49 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:49.659739 16932 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:06:49.665298 16932 webserver.cc:460] Webserver started at http://127.3.240.193:41527/ using document root <none> and password file <none>
I20260927 23:06:49.666306 16932 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:49.666467 16932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 23:06:49 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:49.734355 16932 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.006s sys 0.003s
I20260927 23:06:49.743582 16979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:49.745468 16932 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.005s sys 0.003s
I20260927 23:06:49.745651 16932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "0ab24245470743cd86314e4d2ff43d30"
format_stamp: "Formatted at 2026-09-27 23:06:46 on dist-test-slave-lmfp"
server_key: "0b7d627106562e600cbda138480495cb"
server_key_iv: "258e4a0cde58545004fa764953da195a"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:49.747051 16932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:49.821214 16932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:49.825410 16932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:49.825682 16932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:49.826810 16932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:49.828341 16932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:49.828464 16932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:49.828588 16932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:49.828683 16932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:49.912379 16932 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:41181
I20260927 23:06:49.912420 17091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:41181 every 8 connection(s)
I20260927 23:06:49.914072 16932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:06:49.922097 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 16932
I20260927 23:06:49.922967 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:54791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:46155
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:35221
--webserver_port=37201
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:54791/kms with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:06:49 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550409, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:49.945102 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:49.917492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35221 (local address 127.3.240.193:57797)
0927 23:06:49.918296 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:49.918348 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:49.920448 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:49.922342 (+ 1894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:49.922354 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:49.922992 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:49.923828 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:49.923855 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:49.929120 (+ 5265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:49.929129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:49.930786 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:49.930806 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:49.932264 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:49.932272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:49.932554 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:49.933325 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:49.933360 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:49.935877 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:49.939217 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:49.939225 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:49.939240 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:49.939691 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:49.940168 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:49.940173 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:49.940176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:49.940365 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:49.942273 (+ 1908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:49.942281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:49.942721 (+ 440us) client_negotiation.cc:769] Sending connection context
0927 23:06:49.943076 (+ 355us) client_negotiation.cc:240] Negotiation successful
0927 23:06:49.943662 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":347,"threads_started":1}
I20260927 23:06:49.946695 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:49.917467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57797 (local address 127.3.240.254:35221)
0927 23:06:49.919109 (+ 1642us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:49.919127 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:49.919178 (+ 51us) server_negotiation.cc:407] Connection header received
0927 23:06:49.921125 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:49.921146 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:49.921698 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:49.922966 (+ 1268us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:49.924473 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:49.928902 (+ 4429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:49.930937 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:49.931701 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:49.931886 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:49.936125 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:49.936133 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:49.936147 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:49.936182 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:49.939093 (+ 2911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:49.939879 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:49.939885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:49.939891 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:49.940044 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:49.940491 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:49.940496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:49.940498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:49.941019 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:49.942186 (+ 1167us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:49.943307 (+ 1121us) server_negotiation.cc:299] Negotiation successful
0927 23:06:49.943836 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1346,"thread_start_us":1107,"threads_started":1}
I20260927 23:06:49.951584 17092 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35221
I20260927 23:06:49.951900 17092 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:49.952778 17092 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
W20260927 23:06:49.956059 17092 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:35221 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 23:06:49.956266 17092 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:49.956502 17092 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
I20260927 23:06:49.957746 17092 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:49.958011 17092 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
W20260927 23:06:49.959180 17092 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 23:06:50.331898 17096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:50.332259 17096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:50.332505 17096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:50.344071 17096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:50.344203 17096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:50.344424 17096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
Sep 27 23:06:50 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790550410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:50.359709 17096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:54791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:46155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=37201
--tserver_master_addrs=127.3.240.254:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:50.361346 17096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:50.363035 17096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:50.383219 17113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:50.382874 17112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:50.390108 17115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:50.391880 17096 server_base.cc:1034] running on GCE node
I20260927 23:06:50.392822 17096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:50.407362 17096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:50.408710 17096 hybrid_clock.cc:648] HybridClock initialized: now 1790550410408632 us; error 74 us; skew 500 ppm
I20260927 23:06:50.409114 17096 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:50 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:50.412634 17096 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:06:50.417146 17096 webserver.cc:460] Webserver started at http://127.3.240.194:37201/ using document root <none> and password file <none>
I20260927 23:06:50.417904 17096 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:50.418079 17096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 23:06:50 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:50.442286 17096 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 23:06:50.445771 17123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:50.447448 17096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260927 23:06:50.447713 17096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "788bb0e3dfb442a8954bd403fd567112"
format_stamp: "Formatted at 2026-09-27 23:06:47 on dist-test-slave-lmfp"
server_key: "9910d26246dd2ecb2c82b290d205804d"
server_key_iv: "cf90ed10f0f12cbff1bcd19d8e6d3ebd"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:50.448976 17096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:50.467864 17096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:50.471921 17096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:50.472218 17096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:50.473325 17096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:50.474834 17096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:50.475016 17096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 23:06:50.475164 17096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:50.475281 17096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:50.518512 17096 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:46155
I20260927 23:06:50.518563 17236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:46155 every 8 connection(s)
I20260927 23:06:50.520488 17096 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:06:50.521798 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 17096
I20260927 23:06:50.522574 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:54791/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:42299
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:35221
--webserver_port=36165
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:54791/kms with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:06:50 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:50.612504 17241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:50.551933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:60337 (local address 127.3.240.254:35221)
0927 23:06:50.579643 (+ 27710us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:50.579653 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:50.584127 (+ 4474us) server_negotiation.cc:407] Connection header received
0927 23:06:50.586599 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:50.586607 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:50.586714 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:50.586803 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:50.588698 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:50.589807 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:50.592253 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:50.593041 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:50.593133 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:50.604391 (+ 11258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:50.604400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:50.604405 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:50.604448 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:50.608566 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:50.609511 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:50.609517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:50.609520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:50.609671 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:50.610202 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:50.610207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:50.610210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:50.610594 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:50.610742 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:50.611855 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0927 23:06:50.612090 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27497,"thread_start_us":27734,"threads_started":1}
I20260927 23:06:50.614648 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:50.551286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35221 (local address 127.3.240.194:60337)
0927 23:06:50.580586 (+ 29300us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:50.580657 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:50.585831 (+ 5174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:50.587369 (+ 1538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:50.587377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:50.587826 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:50.588529 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:50.588545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:50.589993 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:50.589998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:50.592091 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:50.592108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:50.593486 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:50.593495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:50.595141 (+ 1646us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:50.600995 (+ 5854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:50.601035 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:50.603956 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:50.608745 (+ 4789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:50.608754 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:50.608770 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:50.609369 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:50.609829 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:50.609834 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:50.609839 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:50.610061 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:50.610838 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:50.610845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:50.611391 (+ 546us) client_negotiation.cc:769] Sending connection context
0927 23:06:50.612809 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0927 23:06:50.613370 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28815,"thread_start_us":32735,"threads_started":1}
I20260927 23:06:50.623719 17237 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35221
I20260927 23:06:50.624107 17237 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:50.625109 17237 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
I20260927 23:06:50.627446 16895 ts_manager.cc:194] Registered new tserver with Master: 788bb0e3dfb442a8954bd403fd567112 (127.3.240.194:46155)
I20260927 23:06:50.630252 16895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:60337
Sep 27 23:06:50 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:50.959347 17092 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:50.959815 17092 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
I20260927 23:06:50.961232 16895 ts_manager.cc:194] Registered new tserver with Master: 0ab24245470743cd86314e4d2ff43d30 (127.3.240.193:41181)
I20260927 23:06:50.962558 16895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:57797
W20260927 23:06:51.033334 17238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:51.033881 17238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:51.034080 17238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:51.045668 17238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:51.045892 17238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:51.046116 17238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:06:51.061550 17238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:54791/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:42299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=36165
--tserver_master_addrs=127.3.240.254:35221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:51.063575 17238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:51.065438 17238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:51.104543 17246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:51.109315 17238 server_base.cc:1034] running on GCE node
W20260927 23:06:51.109297 17247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:51.119323 17249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:51.120272 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:51.124022 17238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:51.125240 17238 hybrid_clock.cc:648] HybridClock initialized: now 1790550411125168 us; error 63 us; skew 500 ppm
I20260927 23:06:51.125638 17238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:51 dist-test-slave-lmfp krb5kdc[15885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550411, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:51.140231 17238 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:06:51.168368 17238 webserver.cc:460] Webserver started at http://127.3.240.195:36165/ using document root <none> and password file <none>
I20260927 23:06:51.169494 17238 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:51.169740 17238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 27 23:06:51 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550411, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:51.205194 17238 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.002s
Sep 27 23:06:51 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550410, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260927 23:06:51.211938 17258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:51.214174 17238 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.004s
I20260927 23:06:51.214394 17238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "edb4af071ce64df7b5fb92fd46a71d5c"
format_stamp: "Formatted at 2026-09-27 23:06:48 on dist-test-slave-lmfp"
server_key: "8982c6a6bc6e76d0e3ea8da6a462ca6c"
server_key_iv: "bf86335ff3c6673f6c2741032418ad0e"
server_key_version: "kuduclusterkey@0"
I20260927 23:06:51.216315 17238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:51.250404 17238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:51.257982 17238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:51.258417 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:51.261629 17238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:06:51.263489 17238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:06:51.263615 17238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:51.263765 17238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:06:51.263856 17238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:51.427562 17238 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42299
I20260927 23:06:51.427587 17370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42299 every 8 connection(s)
I20260927 23:06:51.429796 17238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:06:51.440511 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 17238
Sep 27 23:06:51 dist-test-slave-lmfp krb5kdc[15885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550411, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:51.468714 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.441599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.3.240.254:35221)
0927 23:06:51.452468 (+ 10869us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.452474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.452491 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:06:51.452538 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.452542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.452626 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.452727 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:51.454607 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.455898 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.458151 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.458614 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.458738 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.462588 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:51.462594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:51.462598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:51.462643 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:51.465325 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:51.466084 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:51.466090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:51.466093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:51.466258 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:51.466729 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:51.466735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:51.466738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:51.467060 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:51.467395 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.468275 (+ 880us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.468435 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10757,"thread_start_us":10668,"threads_started":1}
I20260927 23:06:51.477443 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.438799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35221 (local address 127.3.240.195:56419)
0927 23:06:51.447699 (+ 8900us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:51.447756 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:51.450202 (+ 2446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:51.453156 (+ 2954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:51.453169 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:51.453862 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:51.454737 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.454763 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.456326 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.456332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.459348 (+ 3016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.459362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.460635 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.460662 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.461014 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.461759 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:51.461795 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:51.467956 (+ 6161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:51.473726 (+ 5770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:51.473736 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:51.473755 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:51.474180 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:51.474547 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:51.474552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:51.474555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:51.474740 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:51.475329 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:51.475340 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:51.475677 (+ 337us) client_negotiation.cc:769] Sending connection context
0927 23:06:51.475957 (+ 280us) client_negotiation.cc:240] Negotiation successful
0927 23:06:51.476457 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8566,"thread_start_us":8341,"threads_started":1}
I20260927 23:06:51.478499 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.439503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:56419 (local address 127.3.240.254:35221)
0927 23:06:51.451069 (+ 11566us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.451078 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.451122 (+ 44us) server_negotiation.cc:407] Connection header received
0927 23:06:51.451205 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.451214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.451357 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.451486 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:51.454975 (+ 3489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.456188 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.459512 (+ 3324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.460206 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.460303 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.469990 (+ 9687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:51.470000 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:51.470004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:51.470249 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:51.473585 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:51.474307 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:51.474313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:51.474317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:51.474430 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:51.474869 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:51.474878 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:51.474880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:51.475184 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:51.475326 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.478103 (+ 2777us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.478248 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11359,"thread_start_us":11240,"threads_started":1}
I20260927 23:06:51.479622 17371 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35221
I20260927 23:06:51.479929 17371 heartbeater.cc:464] Registering TS with master...
I20260927 23:06:51.480602 17371 heartbeater.cc:511] Master 127.3.240.254:35221 requested a full tablet report, sending...
I20260927 23:06:51.482203 16895 ts_manager.cc:194] Registered new tserver with Master: edb4af071ce64df7b5fb92fd46a71d5c (127.3.240.195:42299)
I20260927 23:06:51.483548 16895 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:56419
I20260927 23:06:51.487401 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:06:51.521365 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.504834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.3.240.254:35221)
0927 23:06:51.505051 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.505059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.505628 (+ 569us) server_negotiation.cc:407] Connection header received
0927 23:06:51.505875 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.505881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.505968 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.506090 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:51.508440 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.509548 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.511416 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.512135 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.512288 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.514914 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:51.514921 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:51.514926 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:51.514966 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:51.518328 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:51.519166 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:51.519172 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:51.519176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:51.519322 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:51.519804 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:51.519809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:51.519812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:51.520180 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:51.520412 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.520790 (+ 378us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.521011 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260927 23:06:51.526669 16895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:06:51.535413 16895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:06:51.578152 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.551815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47314 (local address 127.3.240.193:41181)
0927 23:06:51.565810 (+ 13995us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.565822 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.568403 (+ 2581us) server_negotiation.cc:407] Connection header received
0927 23:06:51.568520 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.568531 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.568832 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.569028 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:51.570272 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.571798 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.574806 (+ 3008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.576367 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.576479 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.576717 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.577671 (+ 954us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.577831 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13763,"thread_start_us":14013,"threads_started":1}
I20260927 23:06:51.578210 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.549164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42299 (local address 127.0.0.1:35266)
0927 23:06:51.567575 (+ 18411us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:51.567634 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:51.567826 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:51.569381 (+ 1555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:51.569385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:51.569405 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:51.570015 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.570031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.572079 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.572084 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.574650 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.574664 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.577032 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.577040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.577226 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.577264 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 23:06:51.577349 (+ 85us) client_negotiation.cc:240] Negotiation successful
0927 23:06:51.577479 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18276,"mutex_wait_us":203,"thread_start_us":18149,"threads_started":1}
I20260927 23:06:51.578392 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.549575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34884 (local address 127.3.240.194:46155)
0927 23:06:51.566508 (+ 16933us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.566520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.568577 (+ 2057us) server_negotiation.cc:407] Connection header received
0927 23:06:51.568697 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.568703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.568899 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.569082 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:51.570277 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.571905 (+ 1628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.575608 (+ 3703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.576983 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.577106 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.577297 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.577943 (+ 646us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.578089 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16754,"thread_start_us":16604,"threads_started":1}
I20260927 23:06:51.578889 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.549536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.3.240.195:42299)
0927 23:06:51.568497 (+ 18961us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.568504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.568535 (+ 31us) server_negotiation.cc:407] Connection header received
0927 23:06:51.568665 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.568672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.568899 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.569083 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:51.570291 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.571905 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.574816 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.576550 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.576681 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.576873 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.578430 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.578601 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18796,"thread_start_us":18634,"threads_started":1}
I20260927 23:06:51.580075 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.551442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41181 (local address 127.0.0.1:47314)
0927 23:06:51.568120 (+ 16678us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:51.568142 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:51.568267 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:51.569292 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:51.569299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:51.569343 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:51.570017 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.570032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.571995 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.572000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.574626 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.574641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.576850 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.576856 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.577041 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.577147 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 23:06:51.577270 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 23:06:51.577450 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16513,"mutex_wait_us":209,"thread_start_us":16362,"threads_started":1}
I20260927 23:06:51.580612 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.549104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46155 (local address 127.0.0.1:34884)
0927 23:06:51.568462 (+ 19358us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:51.568478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:51.568586 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:51.569324 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:51.569328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:51.569355 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:51.570063 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.570074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.572080 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.572084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.575437 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.575452 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.577435 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.577440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.577580 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.577618 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 23:06:51.577716 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 23:06:51.577840 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17731,"mutex_wait_us":126,"thread_start_us":17601,"threads_started":1}
I20260927 23:06:51.582924 17306 tablet_service.cc:1467] Processing CreateTablet for tablet 398f824dd97b43cc91d436f4a453d4dc (DEFAULT_TABLE table=test-table [id=695dbeca23f54cfeaab75e067bcba49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:06:51.584102 17027 tablet_service.cc:1467] Processing CreateTablet for tablet 398f824dd97b43cc91d436f4a453d4dc (DEFAULT_TABLE table=test-table [id=695dbeca23f54cfeaab75e067bcba49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:06:51.584373 17172 tablet_service.cc:1467] Processing CreateTablet for tablet 398f824dd97b43cc91d436f4a453d4dc (DEFAULT_TABLE table=test-table [id=695dbeca23f54cfeaab75e067bcba49d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:06:51.585248 17306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398f824dd97b43cc91d436f4a453d4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:51.585523 17027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398f824dd97b43cc91d436f4a453d4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:51.585690 17172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 398f824dd97b43cc91d436f4a453d4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:51.604212 17401 tablet_bootstrap.cc:492] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Bootstrap starting.
I20260927 23:06:51.610898 17401 tablet_bootstrap.cc:654] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:51.611690 17400 tablet_bootstrap.cc:492] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: Bootstrap starting.
I20260927 23:06:51.612411 17399 tablet_bootstrap.cc:492] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: Bootstrap starting.
I20260927 23:06:51.614938 17399 tablet_bootstrap.cc:654] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:51.616206 17399 log.cc:824] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:51.617908 17400 tablet_bootstrap.cc:654] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:51.618738 17399 tablet_bootstrap.cc:492] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: No bootstrap required, opened a new log
I20260927 23:06:51.619007 17400 log.cc:824] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:51.619040 17399 ts_tablet_manager.cc:1397] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260927 23:06:51.620474 17401 log.cc:824] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:51.623165 17400 tablet_bootstrap.cc:492] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: No bootstrap required, opened a new log
I20260927 23:06:51.623469 17400 ts_tablet_manager.cc:1397] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260927 23:06:51.623593 17401 tablet_bootstrap.cc:492] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: No bootstrap required, opened a new log
I20260927 23:06:51.623881 17401 ts_tablet_manager.cc:1397] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.006s
I20260927 23:06:51.628891 17399 raft_consensus.cc:348] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.629413 17399 raft_consensus.cc:374] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:51.629541 17399 raft_consensus.cc:729] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab24245470743cd86314e4d2ff43d30, State: Initialized, Role: FOLLOWER
I20260927 23:06:51.630281 17399 consensus_queue.cc:260] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.633463 17401 raft_consensus.cc:348] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.633888 17401 raft_consensus.cc:374] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:51.634019 17401 raft_consensus.cc:729] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edb4af071ce64df7b5fb92fd46a71d5c, State: Initialized, Role: FOLLOWER
I20260927 23:06:51.634671 17401 consensus_queue.cc:260] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.634883 17237 heartbeater.cc:503] Master 127.3.240.254:35221 was elected leader, sending a full tablet report...
I20260927 23:06:51.638489 17399 ts_tablet_manager.cc:1428] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: Time spent starting tablet: real 0.019s user 0.009s sys 0.008s
I20260927 23:06:51.638847 17092 heartbeater.cc:503] Master 127.3.240.254:35221 was elected leader, sending a full tablet report...
I20260927 23:06:51.640729 17400 raft_consensus.cc:348] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.641179 17400 raft_consensus.cc:374] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:51.641309 17400 raft_consensus.cc:729] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788bb0e3dfb442a8954bd403fd567112, State: Initialized, Role: FOLLOWER
I20260927 23:06:51.641960 17400 consensus_queue.cc:260] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.644225 17371 heartbeater.cc:503] Master 127.3.240.254:35221 was elected leader, sending a full tablet report...
I20260927 23:06:51.644796 17401 ts_tablet_manager.cc:1428] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
I20260927 23:06:51.645284 17400 ts_tablet_manager.cc:1428] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: Time spent starting tablet: real 0.022s user 0.006s sys 0.008s
W20260927 23:06:51.667083 17093 tablet.cc:2369] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:06:51.702251 17372 tablet.cc:2369] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:06:51.730698 17405 raft_consensus.cc:482] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:06:51.730995 17405 raft_consensus.cc:504] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.733217 17405 leader_election.cc:288] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 788bb0e3dfb442a8954bd403fd567112 (127.3.240.194:46155), edb4af071ce64df7b5fb92fd46a71d5c (127.3.240.195:42299)
I20260927 23:06:51.740849 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.733941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:53563 (local address 127.3.240.194:46155)
0927 23:06:51.734194 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.734203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.734917 (+ 714us) server_negotiation.cc:407] Connection header received
0927 23:06:51.735281 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.735288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.735363 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.735447 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:51.736133 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.737160 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.738911 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.739924 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.740067 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.740139 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.740445 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.740595 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260927 23:06:51.740938 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.733401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46155 (local address 127.3.240.193:53563)
0927 23:06:51.734799 (+ 1398us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:51.734822 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:51.735080 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:51.735614 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:51.735620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:51.735657 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:51.735988 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.735996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.737360 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.737364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.738765 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.738776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.740113 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.740119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.740282 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.740314 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 23:06:51.740407 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 23:06:51.740523 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1237,"mutex_wait_us":41,"thread_start_us":1316,"threads_started":1}
I20260927 23:06:51.741684 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.733858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:49301 (local address 127.3.240.195:42299)
0927 23:06:51.734028 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.734033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.735174 (+ 1141us) server_negotiation.cc:407] Connection header received
0927 23:06:51.735334 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.735339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.735411 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.735494 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:06:51.736176 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.737167 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.739483 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.740588 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.740727 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.740789 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.741353 (+ 564us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.741471 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:06:51.742079 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.733544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42299 (local address 127.3.240.193:49301)
0927 23:06:51.735062 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:51.735079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:51.735205 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:51.735527 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:51.735533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:51.735557 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:06:51.735982 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.735993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.737294 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.737299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.739327 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:51.739341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.740956 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.740962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:51.741183 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.741215 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 23:06:51.741332 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 23:06:51.741725 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1387,"mutex_wait_us":1,"thread_start_us":847,"threads_started":1}
I20260927 23:06:51.742226 17192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398f824dd97b43cc91d436f4a453d4dc" candidate_uuid: "0ab24245470743cd86314e4d2ff43d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "788bb0e3dfb442a8954bd403fd567112" is_pre_election: true
I20260927 23:06:51.742764 17192 raft_consensus.cc:2454] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ab24245470743cd86314e4d2ff43d30 in term 0.
I20260927 23:06:51.743520 16982 leader_election.cc:302] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab24245470743cd86314e4d2ff43d30, 788bb0e3dfb442a8954bd403fd567112; no voters:
I20260927 23:06:51.743593 17326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398f824dd97b43cc91d436f4a453d4dc" candidate_uuid: "0ab24245470743cd86314e4d2ff43d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" is_pre_election: true
I20260927 23:06:51.743896 17405 raft_consensus.cc:2781] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:06:51.744037 17405 raft_consensus.cc:482] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:06:51.744156 17405 raft_consensus.cc:3037] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:51.744091 17326 raft_consensus.cc:2454] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ab24245470743cd86314e4d2ff43d30 in term 0.
I20260927 23:06:51.746424 17405 raft_consensus.cc:504] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.746951 17405 leader_election.cc:288] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [CANDIDATE]: Term 1 election: Requested vote from peers 788bb0e3dfb442a8954bd403fd567112 (127.3.240.194:46155), edb4af071ce64df7b5fb92fd46a71d5c (127.3.240.195:42299)
I20260927 23:06:51.747356 17192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398f824dd97b43cc91d436f4a453d4dc" candidate_uuid: "0ab24245470743cd86314e4d2ff43d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "788bb0e3dfb442a8954bd403fd567112"
I20260927 23:06:51.747408 17326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "398f824dd97b43cc91d436f4a453d4dc" candidate_uuid: "0ab24245470743cd86314e4d2ff43d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edb4af071ce64df7b5fb92fd46a71d5c"
I20260927 23:06:51.747545 17192 raft_consensus.cc:3037] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:51.747578 17326 raft_consensus.cc:3037] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:51.749609 17192 raft_consensus.cc:2454] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ab24245470743cd86314e4d2ff43d30 in term 1.
I20260927 23:06:51.749985 16982 leader_election.cc:302] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ab24245470743cd86314e4d2ff43d30, 788bb0e3dfb442a8954bd403fd567112; no voters:
I20260927 23:06:51.750250 17405 raft_consensus.cc:2781] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:06:51.750569 17326 raft_consensus.cc:2454] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ab24245470743cd86314e4d2ff43d30 in term 1.
I20260927 23:06:51.750789 17405 raft_consensus.cc:686] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 1 LEADER]: Becoming Leader. State: Replica: 0ab24245470743cd86314e4d2ff43d30, State: Running, Role: LEADER
I20260927 23:06:51.751341 17405 consensus_queue.cc:237] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } }
I20260927 23:06:51.756626 16895 catalog_manager.cc:5495] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ab24245470743cd86314e4d2ff43d30 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "0ab24245470743cd86314e4d2ff43d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ab24245470743cd86314e4d2ff43d30" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41181 } health_report { overall_health: HEALTHY } } }
I20260927 23:06:51.799816 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.795234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.3.240.193:41181)
0927 23:06:51.795422 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.795427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.795443 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:06:51.795552 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.795555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.795623 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.795733 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:06:51.796043 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.796698 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.797929 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.798290 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.798388 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.798775 (+ 387us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:06:51.799282 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:06:51.799404 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.799488 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.799545 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 23:06:51.807368 17326 raft_consensus.cc:1261] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 1 FOLLOWER]: Refusing update from remote peer 0ab24245470743cd86314e4d2ff43d30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:06:51.807374 17192 raft_consensus.cc:1261] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 1 FOLLOWER]: Refusing update from remote peer 0ab24245470743cd86314e4d2ff43d30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:06:51.808287 17410 consensus_queue.cc:1035] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edb4af071ce64df7b5fb92fd46a71d5c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:06:51.808620 17405 consensus_queue.cc:1035] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "788bb0e3dfb442a8954bd403fd567112" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:06:51.819705 17413 mvcc.cc:205] Tried to move back new op lower bound from 7334094486753587200 to 7334094486537867264. Current Snapshot: MvccSnapshot[applied={T|T < 7334094486753587200}]
I20260927 23:06:51.832247 16895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:06:51.832530 16895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:06:51.836017 16895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8aec173c4764849b22d1db47b480299: Sending DeleteTablet for 3 replicas of tablet 398f824dd97b43cc91d436f4a453d4dc
I20260927 23:06:51.837033 17172 tablet_service.cc:1514] Processing DeleteTablet for tablet 398f824dd97b43cc91d436f4a453d4dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:06:51 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:34884
I20260927 23:06:51.837064 17027 tablet_service.cc:1514] Processing DeleteTablet for tablet 398f824dd97b43cc91d436f4a453d4dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:06:51 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:47314
I20260927 23:06:51.837057 17306 tablet_service.cc:1514] Processing DeleteTablet for tablet 398f824dd97b43cc91d436f4a453d4dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:06:51 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:35266
I20260927 23:06:51.844677 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 16932
I20260927 23:06:51.859485 17423 tablet_replica.cc:330] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: stopping tablet replica
I20260927 23:06:51.860024 17423 raft_consensus.cc:2229] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:06:51.860601 17423 raft_consensus.cc:2258] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:06:51.865790 17424 tablet_replica.cc:330] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: stopping tablet replica
I20260927 23:06:51.866328 17424 raft_consensus.cc:2229] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:06:51.867146 17424 raft_consensus.cc:2258] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:06:51.867901 17425 tablet_replica.cc:330] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: stopping tablet replica
I20260927 23:06:51.868287 17425 raft_consensus.cc:2229] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:06:51.868727 17425 raft_consensus.cc:2258] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:06:51.869532 17424 ts_tablet_manager.cc:1905] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:06:51.870904 17425 ts_tablet_manager.cc:1905] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:06:51.871197 17423 ts_tablet_manager.cc:1905] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:06:51.876878 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.845792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47328 (local address 127.3.240.193:41181)
0927 23:06:51.845997 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.846005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.846026 (+ 21us) server_negotiation.cc:407] Connection header received
0927 23:06:51.846102 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.846109 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.846212 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.846320 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:51.849421 (+ 3101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.851036 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.853633 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.854482 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.854615 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.858838 (+ 4223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:51.858855 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:51.858878 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:51.858935 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:51.864011 (+ 5076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:51.865032 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:51.865044 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:51.865050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:51.865291 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:51.865799 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:51.865806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:51.865825 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:51.866379 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:51.866768 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:51.876264 (+ 9496us) server_negotiation.cc:299] Negotiation successful
0927 23:06:51.876501 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 23:06:51.877293 17425 ts_tablet_manager.cc:1918] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:06:51.877477 17425 log.cc:1196] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/398f824dd97b43cc91d436f4a453d4dc
I20260927 23:06:51.877646 17423 ts_tablet_manager.cc:1918] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:06:51.877823 17087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:51.877833 17423 log.cc:1196] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/398f824dd97b43cc91d436f4a453d4dc
I20260927 23:06:51.878026 17425 ts_tablet_manager.cc:1939] T 398f824dd97b43cc91d436f4a453d4dc P edb4af071ce64df7b5fb92fd46a71d5c: Deleting consensus metadata
I20260927 23:06:51.878328 17423 ts_tablet_manager.cc:1939] T 398f824dd97b43cc91d436f4a453d4dc P 788bb0e3dfb442a8954bd403fd567112: Deleting consensus metadata
I20260927 23:06:51.880227 16881 catalog_manager.cc:4841] TS 788bb0e3dfb442a8954bd403fd567112 (127.3.240.194:46155): tablet 398f824dd97b43cc91d436f4a453d4dc (table test-table [id=695dbeca23f54cfeaab75e067bcba49d]) successfully deleted
I20260927 23:06:51.880450 16880 catalog_manager.cc:4841] TS edb4af071ce64df7b5fb92fd46a71d5c (127.3.240.195:42299): tablet 398f824dd97b43cc91d436f4a453d4dc (table test-table [id=695dbeca23f54cfeaab75e067bcba49d]) successfully deleted
I20260927 23:06:51.972146 17424 ts_tablet_manager.cc:1918] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:06:51.972486 17424 log.cc:1196] T 398f824dd97b43cc91d436f4a453d4dc P 0ab24245470743cd86314e4d2ff43d30: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/398f824dd97b43cc91d436f4a453d4dc
I20260927 23:06:51.972819 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 16932
W20260927 23:06:51.989588 16882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:41181 (error 108)
I20260927 23:06:51.989838 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 17096
I20260927 23:06:51.991245 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.990797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41181 (local address 127.0.0.1:47344)
0927 23:06:51.990947 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:51.991125 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260927 23:06:51.991513 16882 catalog_manager.cc:4584] TS 0ab24245470743cd86314e4d2ff43d30 (127.3.240.193:41181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 398f824dd97b43cc91d436f4a453d4dc: Network error: Client connection negotiation failed: client connection to 127.3.240.193:41181: connect: Connection refused (error 111)
I20260927 23:06:52.004137 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:51.990605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.3.240.194:46155)
0927 23:06:51.990799 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:51.990805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:51.990929 (+ 124us) server_negotiation.cc:407] Connection header received
0927 23:06:51.991228 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:51.991238 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:51.991311 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:51.991441 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:51.993299 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.994111 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.995679 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:51.996134 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:51.996294 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:51.998732 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:51.998745 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:51.998760 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:51.998794 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:52.001414 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:52.002083 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:52.002098 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:52.002102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:52.002232 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:52.002636 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:52.002641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:52.002646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:52.002963 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:52.003356 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:52.003591 (+ 235us) server_negotiation.cc:299] Negotiation successful
0927 23:06:52.003821 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:06:52.004720 17232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:52.045101 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:52.044532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41181 (local address 127.0.0.1:47348)
0927 23:06:52.044769 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:52.044894 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 23:06:52.046120 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:52.045757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41181 (local address 127.0.0.1:47352)
0927 23:06:52.045899 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:52.045992 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 23:06:52.092211 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 17096
I20260927 23:06:52.106776 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:52.106205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41181 (local address 127.0.0.1:47360)
0927 23:06:52.106461 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:52.106577 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 23:06:52.107904 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:52.107484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41181 (local address 127.0.0.1:47362)
0927 23:06:52.107661 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:52.107749 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 23:06:52.109529 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 17238
I20260927 23:06:52.121939 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:52.110285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35268 (local address 127.3.240.195:42299)
0927 23:06:52.110461 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:52.110465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:52.110481 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:06:52.110537 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:52.110540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:52.110612 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:52.110702 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:52.112010 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:52.112779 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:52.114085 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:52.114498 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:52.114608 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:52.117047 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:52.117062 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:52.117079 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:52.117108 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:52.119711 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:52.120286 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:52.120292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:52.120294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:52.120413 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:52.120701 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:52.120704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:52.120706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:52.121014 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:52.121251 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:52.121520 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 23:06:52.121688 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:06:52.122534 17366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:52.188467 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:52.187893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41181 (local address 127.0.0.1:47366)
0927 23:06:52.188126 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:52.188240 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260927 23:06:52.189620 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:52.189169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41181 (local address 127.0.0.1:47374)
0927 23:06:52.189396 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:52.189485 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:41181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260927 23:06:52.210749 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 17238
I20260927 23:06:52.226336 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 16862
I20260927 23:06:52.231869 16925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:52.339497 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 16862
I20260927 23:06:52.363173 4035 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260927 23:06:52.698554 4035 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260927 23:06:52.698727 4035 mini_ranger.cc:66] Stopping Ranger...
I20260927 23:06:53.043093 4035 mini_ranger.cc:68] Stopped Ranger
2026-09-27 23:06:53.043 UTC [15915] LOG: received smart shutdown request
2026-09-27 23:06:53.045 UTC [15915] LOG: background worker "logical replication launcher" (PID 15924) exited with exit code 1
2026-09-27 23:06:53.046 UTC [15919] LOG: shutting down
2026-09-27 23:06:53.204 UTC [15915] LOG: database system is shut down
2026-09-27T23:06:53Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48977 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:06:53 dist-test-slave-lmfp krb5kdc[17439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:06:53 dist-test-slave-lmfp krb5kdc[17439](info): set up 2 sockets
Sep 27 23:06:53 dist-test-slave-lmfp krb5kdc[17439](info): commencing operation
krb5kdc: starting...
W20260927 23:06:54.423288 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:06:54 dist-test-slave-lmfp krb5kdc[17439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:06:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:06:54.636507 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40233
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:40233,127.3.240.253:33311,127.3.240.252:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:06:54.947670 17452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:54.947952 17452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:54.948046 17452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:54.954828 17452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:06:54.954919 17452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:54.954972 17452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:54.955078 17452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:06:54.955130 17452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:06:54.964262 17452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:40233,127.3.240.253:33311,127.3.240.252:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:54.965346 17452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:54.966460 17452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:54.977329 17458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:54.977203 17460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:54.977615 17457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:54.978329 17452 server_base.cc:1034] running on GCE node
I20260927 23:06:54.978929 17452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:54.980070 17452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:54.981254 17452 hybrid_clock.cc:648] HybridClock initialized: now 1790550414981218 us; error 31 us; skew 500 ppm
I20260927 23:06:54.981511 17452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:54 dist-test-slave-lmfp krb5kdc[17439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550414, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:54.984510 17452 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:06:54.985760 17452 webserver.cc:460] Webserver started at http://127.3.240.254:43643/ using document root <none> and password file <none>
I20260927 23:06:54.986382 17452 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:54.986517 17452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:54.986819 17452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:06:54.989547 17452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "55f0c80e9afb477d9af773c8d635d759"
format_stamp: "Formatted at 2026-09-27 23:06:54 on dist-test-slave-lmfp"
server_key: "b42bcbfd193daa5b4d9a7f2fcd22d8d2"
server_key_iv: "8052df47e83047fc6ef9b044193e36a9"
server_key_version: "encryptionkey@0"
I20260927 23:06:54.990305 17452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "55f0c80e9afb477d9af773c8d635d759"
format_stamp: "Formatted at 2026-09-27 23:06:54 on dist-test-slave-lmfp"
server_key: "b42bcbfd193daa5b4d9a7f2fcd22d8d2"
server_key_iv: "8052df47e83047fc6ef9b044193e36a9"
server_key_version: "encryptionkey@0"
I20260927 23:06:54.994891 17452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 23:06:54.998140 17467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:54.999464 17452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:06:54.999634 17452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "55f0c80e9afb477d9af773c8d635d759"
format_stamp: "Formatted at 2026-09-27 23:06:54 on dist-test-slave-lmfp"
server_key: "b42bcbfd193daa5b4d9a7f2fcd22d8d2"
server_key_iv: "8052df47e83047fc6ef9b044193e36a9"
server_key_version: "encryptionkey@0"
I20260927 23:06:54.999842 17452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:55.020910 17452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:55.024596 17452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:55.024849 17452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:55.041937 17452 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:40233
I20260927 23:06:55.041978 17518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:40233 every 8 connection(s)
I20260927 23:06:55.043210 17452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:06:55.045589 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 17452
I20260927 23:06:55.045811 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:06:55.046574 4035 external_mini_cluster.cc:1428] Setting key 9e01e1d73317807167b05505e708f2f8
I20260927 23:06:55.046914 17519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:55.055809 17519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.240.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.253@KRBTEST.COM" created.
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550414, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:55.077313 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.058793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40233 (local address 127.0.0.1:43966)
0927 23:06:55.059899 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.059931 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:55.061661 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:55.062526 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:55.062533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:55.063323 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:55.063745 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.063764 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.065272 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.065277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.066300 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.066317 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.067209 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.067215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.067373 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.067970 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:55.067995 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:55.070100 (+ 2105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:55.073080 (+ 2980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.073091 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.073102 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.073465 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.073847 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.073852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.073856 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.073994 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.074853 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:55.074860 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:55.075123 (+ 263us) client_negotiation.cc:769] Sending connection context
0927 23:06:55.075377 (+ 254us) client_negotiation.cc:240] Negotiation successful
0927 23:06:55.075986 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":200,"spinlock_wait_cycles":22784,"thread_start_us":701,"threads_started":1}
I20260927 23:06:55.077342 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.058785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43966 (local address 127.3.240.254:40233)
0927 23:06:55.059638 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.059653 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.060051 (+ 398us) server_negotiation.cc:407] Connection header received
0927 23:06:55.062037 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.062042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.062324 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.062521 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.063938 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.065112 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.066433 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.066941 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.067180 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.070300 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:55.070308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:55.070319 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:55.070351 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:55.072961 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.073582 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.073594 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.073596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.073739 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.074097 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.074103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.074106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.074707 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:55.074930 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:55.075468 (+ 538us) server_negotiation.cc:299] Negotiation successful
0927 23:06:55.075882 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":606,"thread_start_us":389,"threads_started":1}
I20260927 23:06:55.080207 17519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:06:55.081149 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.080816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53918)
0927 23:06:55.080953 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.081047 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 23:06:55.081388 17470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
I20260927 23:06:55.082975 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.082616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53930)
0927 23:06:55.082781 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.082847 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260927 23:06:55.083280 17519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33311: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
I20260927 23:06:55.093801 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:33311
--webserver_interface=127.3.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:40233,127.3.240.253:33311,127.3.240.252:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:06:55.132661 17519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } attempt: 1
I20260927 23:06:55.133808 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.133362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53944)
0927 23:06:55.133550 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.133672 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 23:06:55.134850 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.134523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53958)
0927 23:06:55.134646 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.134715 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 23:06:55.135128 17519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33311: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
I20260927 23:06:55.203418 17519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } attempt: 2
I20260927 23:06:55.204749 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.204154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53960)
0927 23:06:55.204504 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.204606 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 23:06:55.205685 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.205332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53970)
0927 23:06:55.205501 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.205566 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260927 23:06:55.205835 17519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33311: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
I20260927 23:06:55.297073 17519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } attempt: 3
I20260927 23:06:55.298293 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.297716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53984)
0927 23:06:55.297980 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.298097 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 23:06:55.299554 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.299035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53986)
0927 23:06:55.299219 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.299338 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260927 23:06:55.299731 17519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33311: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
W20260927 23:06:55.408008 17526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:55.408257 17526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:55.408397 17526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:55.414919 17526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:06:55.415042 17526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:55.415110 17526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:55.415182 17526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:06:55.415223 17526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:06:55.424309 17526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:40233,127.3.240.253:33311,127.3.240.252:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:55.425469 17526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:55.426620 17526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:55.437567 17534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:55.437525 17531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:55.437647 17532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:55.438146 17526 server_base.cc:1034] running on GCE node
I20260927 23:06:55.438864 17526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:55.440093 17526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:55.441305 17526 hybrid_clock.cc:648] HybridClock initialized: now 1790550415441267 us; error 50 us; skew 500 ppm
I20260927 23:06:55.441618 17526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:06:55.442966 17519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } attempt: 4
I20260927 23:06:55.444105 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.443592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53990)
0927 23:06:55.443867 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.443950 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:55.444975 17526 init.cc:377] Logged in from keytab as kudu/127.3.240.253@KRBTEST.COM (short username kudu)
I20260927 23:06:55.445113 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.444706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:53998)
0927 23:06:55.444864 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.444950 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 23:06:55.445356 17519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33311: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33311: connect: Connection refused (error 111)
I20260927 23:06:55.446314 17526 webserver.cc:460] Webserver started at http://127.3.240.253:34905/ using document root <none> and password file <none>
I20260927 23:06:55.446987 17526 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:55.447158 17526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:55.447441 17526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:06:55.450053 17526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/data/instance:
uuid: "02412a9de0b74aadabf856f7fc27ba4c"
format_stamp: "Formatted at 2026-09-27 23:06:55 on dist-test-slave-lmfp"
server_key: "74c218dc2f2cebe41e61ea0868363b69"
server_key_iv: "295e929e862336e0e169c70f07467b78"
server_key_version: "encryptionkey@0"
I20260927 23:06:55.450819 17526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/wal/instance:
uuid: "02412a9de0b74aadabf856f7fc27ba4c"
format_stamp: "Formatted at 2026-09-27 23:06:55 on dist-test-slave-lmfp"
server_key: "74c218dc2f2cebe41e61ea0868363b69"
server_key_iv: "295e929e862336e0e169c70f07467b78"
server_key_version: "encryptionkey@0"
I20260927 23:06:55.455288 17526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:06:55.458302 17541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:55.459589 17526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:06:55.459760 17526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/wal
uuid: "02412a9de0b74aadabf856f7fc27ba4c"
format_stamp: "Formatted at 2026-09-27 23:06:55 on dist-test-slave-lmfp"
server_key: "74c218dc2f2cebe41e61ea0868363b69"
server_key_iv: "295e929e862336e0e169c70f07467b78"
server_key_version: "encryptionkey@0"
I20260927 23:06:55.459954 17526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:55.485109 17526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:55.488592 17526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:55.488867 17526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:55.506379 17526 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:33311
I20260927 23:06:55.506412 17592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:33311 every 8 connection(s)
I20260927 23:06:55.507762 17526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
I20260927 23:06:55.511222 17593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:55.514870 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 17526
I20260927 23:06:55.515089 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-1/wal/instance
I20260927 23:06:55.515609 4035 external_mini_cluster.cc:1428] Setting key 5ee832f60506c1ce344bc022421c1143
I20260927 23:06:55.519125 17593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.252@KRBTEST.COM" created.
I20260927 23:06:55.539649 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.522579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.3.240.254:40233)
0927 23:06:55.522811 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.522820 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.524056 (+ 1236us) server_negotiation.cc:407] Connection header received
0927 23:06:55.525878 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.525883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.525967 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.526072 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.527567 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.528171 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.529658 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.530078 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.530198 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.533517 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:55.533522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:55.533525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:55.533556 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:55.536313 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.537077 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.537082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.537084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.537189 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.537724 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.537729 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.537732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.538087 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:55.538233 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:55.538934 (+ 701us) server_negotiation.cc:299] Negotiation successful
0927 23:06:55.539201 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:06:55.540226 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.523039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40233 (local address 127.0.0.1:43980)
0927 23:06:55.523878 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.523921 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:55.525599 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:55.526287 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:55.526294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:55.526841 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:55.527417 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.527433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.528320 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.528323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.529506 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.529518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.530301 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.530308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.530511 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.531205 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:55.531226 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:55.533321 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:55.536444 (+ 3123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.536452 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.536467 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.536957 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.537359 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.537367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.537372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.537593 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.538220 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:55.538226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:55.538553 (+ 327us) client_negotiation.cc:769] Sending connection context
0927 23:06:55.538776 (+ 223us) client_negotiation.cc:240] Negotiation successful
0927 23:06:55.539181 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":351,"threads_started":1}
I20260927 23:06:55.541896 17593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:06:55.557016 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.542488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:54012)
0927 23:06:55.542614 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.542633 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:55.542727 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:55.543776 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:55.543782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:55.544221 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:55.544425 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.544439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.545615 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.545622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.547068 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.547084 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.548338 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.548346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.548584 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.549124 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:55.549149 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:55.551455 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:55.554585 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.554591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.554598 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.554947 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.555347 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.555351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.555355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.555496 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.556053 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:55.556058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:55.556199 (+ 141us) client_negotiation.cc:769] Sending connection context
0927 23:06:55.556361 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 23:06:55.556628 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260927 23:06:55.557125 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.542727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.3.240.253:33311)
0927 23:06:55.543390 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.543397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.543424 (+ 27us) server_negotiation.cc:407] Connection header received
0927 23:06:55.543470 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.543475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.543624 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.543723 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.544569 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.545488 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.547242 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.548173 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.548264 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.551650 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:55.551662 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:55.551674 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:55.551706 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:55.554456 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.555114 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.555126 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.555128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.555230 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.555602 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.555606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.555609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.555940 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:55.556148 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:55.556511 (+ 363us) server_negotiation.cc:299] Negotiation successful
0927 23:06:55.556732 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":467,"thread_start_us":312,"threads_started":1}
I20260927 23:06:55.559527 17593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:06:55.560696 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.560273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54716)
0927 23:06:55.560407 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.560535 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 23:06:55.561013 17545 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
I20260927 23:06:55.562572 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.562154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54728)
0927 23:06:55.562347 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.562469 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
W20260927 23:06:55.562824 17593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39305: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
I20260927 23:06:55.563998 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:39305
--webserver_interface=127.3.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:40233,127.3.240.253:33311,127.3.240.252:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:06:55.612155 17593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } attempt: 1
I20260927 23:06:55.613291 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.612848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54730)
0927 23:06:55.613036 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.613133 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 23:06:55.614248 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.613926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54742)
0927 23:06:55.614082 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.614144 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 23:06:55.614475 17593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39305: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
I20260927 23:06:55.682680 17593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } attempt: 2
I20260927 23:06:55.683761 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.683338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54748)
0927 23:06:55.683529 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.683618 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 23:06:55.684685 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.684391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54754)
0927 23:06:55.684531 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.684587 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 23:06:55.684859 17593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39305: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
I20260927 23:06:55.744633 17519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } attempt: 5
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550414, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:06:55.759243 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.745365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:54022)
0927 23:06:55.745550 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.745572 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:55.745687 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:55.746089 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:55.746092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:55.746434 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:55.746601 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.746609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.747990 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.747993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.748836 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.748848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.749767 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.749771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.749908 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.750523 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:55.750541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:55.752774 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:55.757000 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.757004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.757006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.757368 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.757736 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.757739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.757742 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.757863 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.758465 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:55.758468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:55.758588 (+ 120us) client_negotiation.cc:769] Sending connection context
0927 23:06:55.758691 (+ 103us) client_negotiation.cc:240] Negotiation successful
0927 23:06:55.758877 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260927 23:06:55.759395 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.745459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.3.240.253:33311)
0927 23:06:55.745642 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.745648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.745671 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:06:55.745855 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.745861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.745942 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.746059 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.746758 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.747873 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.748959 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.749631 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.749779 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.752995 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:55.753005 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:55.753011 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:55.753059 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:55.756857 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.757480 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.757484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.757488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.757628 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.757959 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.757966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.757971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.758348 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:55.758512 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:55.758822 (+ 310us) server_negotiation.cc:299] Negotiation successful
0927 23:06:55.759090 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 23:06:55.760330 17519 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:06:55.761325 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.761003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54768)
0927 23:06:55.761143 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.761210 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260927 23:06:55.762197 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.761909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54774)
0927 23:06:55.762056 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.762113 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260927 23:06:55.762377 17519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39305: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
I20260927 23:06:55.776027 17593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } attempt: 3
I20260927 23:06:55.777050 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.776511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54788)
0927 23:06:55.776721 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.776859 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 23:06:55.778120 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.777646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54796)
0927 23:06:55.777883 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.777985 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260927 23:06:55.778381 17593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39305: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
I20260927 23:06:55.813580 17519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } attempt: 1
I20260927 23:06:55.814692 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.814252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54810)
0927 23:06:55.814440 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.814540 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 23:06:55.815740 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.815440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54826)
0927 23:06:55.815570 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.815645 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 23:06:55.815964 17519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39305: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
W20260927 23:06:55.872611 17600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:55.872884 17600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:55.873024 17600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:55.879680 17600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:06:55.879765 17600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:55.879855 17600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:55.879940 17600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:06:55.880007 17600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:06:55.884174 17519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } attempt: 2
I20260927 23:06:55.885221 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.884760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54842)
0927 23:06:55.884961 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.885066 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 23:06:55.886307 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.885902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54854)
0927 23:06:55.886071 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.886163 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260927 23:06:55.886554 17519 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39305: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
I20260927 23:06:55.889324 17600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:40233,127.3.240.253:33311,127.3.240.252:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:55.890528 17600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:55.891824 17600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:55.902900 17605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:55.902967 17606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:55.903142 17608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:55.903702 17600 server_base.cc:1034] running on GCE node
I20260927 23:06:55.904424 17600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:55.905643 17600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:55.906874 17600 hybrid_clock.cc:648] HybridClock initialized: now 1790550415906819 us; error 56 us; skew 500 ppm
I20260927 23:06:55.907289 17600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:55.911602 17600 init.cc:377] Logged in from keytab as kudu/127.3.240.252@KRBTEST.COM (short username kudu)
I20260927 23:06:55.913298 17600 webserver.cc:460] Webserver started at http://127.3.240.252:36121/ using document root <none> and password file <none>
I20260927 23:06:55.914191 17600 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:55.914315 17600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:55.914654 17600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:06:55.917583 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/data/instance:
uuid: "45adb01979b74d56892f7df2d9748f64"
format_stamp: "Formatted at 2026-09-27 23:06:55 on dist-test-slave-lmfp"
server_key: "001ba2766946a4336fa450803ddb672b"
server_key_iv: "bf91f5d40901693f62fd9a4d2a50d711"
server_key_version: "encryptionkey@0"
I20260927 23:06:55.918309 17600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/wal/instance:
uuid: "45adb01979b74d56892f7df2d9748f64"
format_stamp: "Formatted at 2026-09-27 23:06:55 on dist-test-slave-lmfp"
server_key: "001ba2766946a4336fa450803ddb672b"
server_key_iv: "bf91f5d40901693f62fd9a4d2a50d711"
server_key_version: "encryptionkey@0"
I20260927 23:06:55.921586 17593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } attempt: 4
I20260927 23:06:55.922539 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.922104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54856)
0927 23:06:55.922322 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.922395 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 23:06:55.923396 17600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260927 23:06:55.923591 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.923241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54870)
0927 23:06:55.923389 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.923452 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 23:06:55.923820 17593 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39305: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39305: connect: Connection refused (error 111)
I20260927 23:06:55.926810 17615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:55.928052 17600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:06:55.928212 17600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/wal
uuid: "45adb01979b74d56892f7df2d9748f64"
format_stamp: "Formatted at 2026-09-27 23:06:55 on dist-test-slave-lmfp"
server_key: "001ba2766946a4336fa450803ddb672b"
server_key_iv: "bf91f5d40901693f62fd9a4d2a50d711"
server_key_version: "encryptionkey@0"
I20260927 23:06:55.928385 17600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:55.940433 17600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:55.944139 17600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:55.944391 17600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:55.961408 17600 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:39305
I20260927 23:06:55.961445 17666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:39305 every 8 connection(s)
I20260927 23:06:55.962677 17600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
I20260927 23:06:55.966215 17667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:55.972290 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 17600
I20260927 23:06:55.972463 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790550224434615-4035-0/minicluster-data/master-2/wal/instance
I20260927 23:06:55.971848 17667 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:06:55.972960 4035 external_mini_cluster.cc:1428] Setting key 2a31885c436c8e19458e7aaa17f14d01
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260927 23:06:55.991840 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.975553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.3.240.254:40233)
0927 23:06:55.975793 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.975799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.976275 (+ 476us) server_negotiation.cc:407] Connection header received
0927 23:06:55.977490 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.977494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.977552 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.977630 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.978931 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.979648 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.981308 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.981936 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.982030 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.985533 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:55.985538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:55.985541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:55.985573 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:55.988746 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.989439 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.989444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.989447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.989617 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.989995 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.990003 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.990006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.990328 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:55.990436 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:55.991276 (+ 840us) server_negotiation.cc:299] Negotiation successful
0927 23:06:55.991438 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260927 23:06:55.992735 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.975451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40233 (local address 127.0.0.1:43994)
0927 23:06:55.976138 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.976181 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:55.977255 (+ 1074us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:55.977710 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:55.977716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:55.978399 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:55.978747 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.978763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.979810 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.979816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.981186 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.981197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.982127 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.982134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.982487 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.983246 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:55.983265 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:55.985356 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:55.988871 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.988882 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.988889 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.989331 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.989718 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:55.989721 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:55.989723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:55.989891 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:55.990451 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:55.990460 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:55.990862 (+ 402us) client_negotiation.cc:769] Sending connection context
0927 23:06:55.991172 (+ 310us) client_negotiation.cc:240] Negotiation successful
0927 23:06:55.991634 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":27,"thread_start_us":263,"threads_started":1}
I20260927 23:06:55.992764 17519 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } attempt: 3
I20260927 23:06:55.994215 17670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.974577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.3.240.252:39305)
0927 23:06:55.975244 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.975253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.975310 (+ 57us) server_negotiation.cc:407] Connection header received
0927 23:06:55.977255 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.977279 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.977537 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.977712 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.979068 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.980513 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.981897 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.982692 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.982824 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:55.986870 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:55.986881 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:55.986895 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:55.986934 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:55.990187 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.990932 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.990938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.990940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.991156 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:55.991551 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:55.991556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:55.991559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:55.991965 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:55.992206 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:55.992705 (+ 499us) server_negotiation.cc:299] Negotiation successful
0927 23:06:55.993750 (+ 1045us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":490,"spinlock_wait_cycles":12032,"thread_start_us":257,"threads_started":1}
I20260927 23:06:55.994736 17667 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550414, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Sep 27 23:06:55 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
Sep 27 23:06:56 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:56.009152 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.993436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.3.240.252:39305)
0927 23:06:55.994123 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.994128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.994155 (+ 27us) server_negotiation.cc:407] Connection header received
0927 23:06:55.994215 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.994219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.994287 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.994445 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.995301 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.996899 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.998751 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.999420 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.999544 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.003223 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.003230 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.003239 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.003276 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.006386 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.007152 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.007156 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.007158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.007267 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.007658 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.007663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.007666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.008017 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.008160 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.008577 (+ 417us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.008779 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":553,"thread_start_us":361,"threads_started":1}
I20260927 23:06:56.009675 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.993251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54898)
0927 23:06:55.993453 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.993472 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:55.993564 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:55.994488 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:55.994493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:55.994927 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:55.995139 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.995152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.997055 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.997061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.998590 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.998604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.999558 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.999564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.999740 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.000321 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:56.000349 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:56.002965 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:56.006525 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:56.006530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:56.006534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:56.007047 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:56.007369 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:56.007375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:56.007380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:56.007545 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:56.008154 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:56.008159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:56.008327 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 23:06:56.008471 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 23:06:56.009434 (+ 963us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 23:06:56.010139 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.995616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33311 (local address 127.0.0.1:54034)
0927 23:06:55.995746 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:55.995766 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:55.995851 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:55.996134 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:55.996139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:55.996639 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:55.996843 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.996852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.998126 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.998132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:55.999420 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:55.999434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.000398 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.000416 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:56.000633 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.001162 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:56.001181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:56.003619 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:56.007543 (+ 3924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:56.007546 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:56.007548 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:56.008048 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:56.008474 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:56.008479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:56.008482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:56.008680 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:56.009218 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:56.009224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:56.009398 (+ 174us) client_negotiation.cc:769] Sending connection context
0927 23:06:56.009558 (+ 160us) client_negotiation.cc:240] Negotiation successful
0927 23:06:56.009787 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260927 23:06:56.010176 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.995680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.3.240.253:33311)
0927 23:06:55.995841 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.995845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.995861 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:06:55.995947 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.995951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.996002 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.996105 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.997004 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:55.997973 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:55.999588 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.000233 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.000366 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.003840 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.003846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.003853 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.003889 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.007432 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.008178 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.008184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.008187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.008341 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.008780 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.008793 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.008796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.009086 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.009362 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.009661 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.009848 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:06:56.010986 17667 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:06:56.015753 17519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: Bootstrap starting.
I20260927 23:06:56.016278 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:55.997418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44000 (local address 127.3.240.254:40233)
0927 23:06:55.997589 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:55.997594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:55.997609 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:06:55.997652 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:55.997656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:55.997734 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:55.997838 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:55.999140 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.000253 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.001643 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.002043 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.002185 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.008476 (+ 6291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.008481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.008486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.008518 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.012848 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.013662 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.013667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.013671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.013813 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.014183 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.014188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.014191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.014566 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.014857 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.015725 (+ 868us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.015940 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Sep 27 23:06:56 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260927 23:06:56.019770 17519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:56.020939 17519 log.cc:824] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:56.023852 17519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: No bootstrap required, opened a new log
I20260927 23:06:56.025457 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:56.011719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54914)
0927 23:06:56.011893 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:56.011911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:56.012009 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:56.012444 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:56.012449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:56.012859 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:56.013050 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:56.013059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.013903 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.013911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:56.015204 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:56.015219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.016010 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.016016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:56.016204 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.016669 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:56.016684 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:56.018990 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:56.022655 (+ 3665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:56.022664 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:56.022669 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:56.023154 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:56.023561 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:56.023566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:56.023569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:56.023728 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:56.024400 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:56.024413 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:56.024586 (+ 173us) client_negotiation.cc:769] Sending connection context
0927 23:06:56.024764 (+ 178us) client_negotiation.cc:240] Negotiation successful
0927 23:06:56.025060 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 23:06:56.025503 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:56.012016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.3.240.252:39305)
0927 23:06:56.012156 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:56.012161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:56.012175 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:06:56.012218 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:56.012222 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:56.012281 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:56.012398 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:56.013169 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.013784 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.015359 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.015890 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.015996 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.019186 (+ 3190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.019195 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.019201 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.019237 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.022528 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.023279 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.023295 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.023298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.023444 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.023863 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.023871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.023876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.024252 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.024432 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.024861 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.025190 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":42}
Sep 27 23:06:56 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:06:56.030345 17667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: Bootstrap starting.
I20260927 23:06:56.033691 17667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:56.034099 17519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.034482 17519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:56.034613 17519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55f0c80e9afb477d9af773c8d635d759, State: Initialized, Role: FOLLOWER
I20260927 23:06:56.034770 17667 log.cc:824] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:56.035357 17519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.036818 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:56.018222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.3.240.253:33311)
0927 23:06:56.018479 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:56.018485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:56.018504 (+ 19us) server_negotiation.cc:407] Connection header received
0927 23:06:56.018557 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:56.018562 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:56.018629 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:56.018723 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:56.020436 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.021552 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.023232 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.023753 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.023865 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.029233 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.029240 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.029244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.029300 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.032930 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.033799 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.033804 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.033808 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.033992 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.034434 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.034438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.034441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.034794 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.035082 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.036247 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.036471 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:06:56.037492 17519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:06:56.037786 17667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: No bootstrap required, opened a new log
I20260927 23:06:56.040069 17675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.040732 17675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:06:56.042289 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:06:56.049268 17667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.049697 17667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:56.049835 17667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45adb01979b74d56892f7df2d9748f64, State: Initialized, Role: FOLLOWER
I20260927 23:06:56.050640 17667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.053133 17681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.053532 17681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:06:56.054678 17667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 23:06:56.074482 17694 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:06:56.074668 17694 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:06:56.069437 17452 master_runner.cc:186] Error getting master registration for 127.3.240.253:33311: OK, instance_id { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" instance_seqno: 1790550415501309 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 23:06:56.081156 17452 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 23:06:56.094180 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:56.070875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.3.240.254:40233)
0927 23:06:56.071096 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:56.071107 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:56.075374 (+ 4267us) server_negotiation.cc:407] Connection header received
0927 23:06:56.075589 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:56.075597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:56.075695 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:56.075792 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:56.078048 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.079195 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.081733 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.082498 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.082594 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.086712 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.086720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.086725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.086807 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.090545 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.091436 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.091442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.091445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.091599 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.092017 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.092022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.092025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.092380 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.092631 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.093607 (+ 976us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.093829 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 23:06:56.101033 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:56.071072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54920 (local address 127.3.240.252:39305)
0927 23:06:56.071253 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:56.071259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:56.076030 (+ 4771us) server_negotiation.cc:407] Connection header received
0927 23:06:56.076267 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:56.076272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:56.076474 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:56.076539 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:56.078427 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.079389 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.081706 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.082595 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.082735 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.087096 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.087104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.087112 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.087157 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.093591 (+ 6434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.094570 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.094579 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.094584 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.096398 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.096948 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.096955 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.096960 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.097331 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.097560 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.100419 (+ 2859us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.100646 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260927 23:06:56.105621 17706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:06:56.105736 17706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:06:56.110534 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:56.075471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54062 (local address 127.3.240.253:33311)
0927 23:06:56.075654 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:56.075661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:56.087556 (+ 11895us) server_negotiation.cc:407] Connection header received
0927 23:06:56.087813 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:56.087818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:56.087915 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:56.087988 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:56.089531 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.090570 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.096593 (+ 6023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.097260 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.097353 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.100743 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.100750 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.100755 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.100809 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.106563 (+ 5754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.107512 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.107521 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.107526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.107695 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.108183 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.108190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.108195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.108575 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.108804 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.109882 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.110106 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:06:56.112188 17600 master_runner.cc:186] Error getting master registration for 127.3.240.253:33311: OK, instance_id { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" instance_seqno: 1790550415501309 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 23:06:56.129916 17600 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260927 23:06:56.196260 17621 tablet.cc:2369] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:06:56.223073 17593 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } attempt: 5
Sep 27 23:06:56 dist-test-slave-lmfp krb5kdc[17439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550415, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260927 23:06:56.243436 17672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:56.223755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54934 (local address 127.3.240.252:39305)
0927 23:06:56.223918 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:56.223925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:56.223998 (+ 73us) server_negotiation.cc:407] Connection header received
0927 23:06:56.224145 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:56.224151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:56.224227 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:56.224348 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:56.225269 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.226432 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.228174 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.228930 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.229066 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.236695 (+ 7629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:56.236702 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:56.236707 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:56.236754 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:56.240508 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.241331 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.241337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.241341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.241480 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:56.241919 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:56.241924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:56.241928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:56.242313 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:56.242459 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:56.242863 (+ 404us) server_negotiation.cc:299] Negotiation successful
0927 23:06:56.243092 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260927 23:06:56.243443 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:56.223646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39305 (local address 127.0.0.1:54934)
0927 23:06:56.223898 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0927 23:06:56.223917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:06:56.224021 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:06:56.224392 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:06:56.224397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:06:56.224919 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:06:56.225118 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:56.225130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.226568 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.226575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:56.228008 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:06:56.228022 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:06:56.229086 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:06:56.229093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:06:56.229308 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:56.233459 (+ 4151us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:06:56.233488 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:06:56.236482 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:06:56.240661 (+ 4179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:56.240667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:56.240672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:56.241217 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:56.241607 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:06:56.241613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:06:56.241617 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:06:56.241800 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:06:56.242438 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:06:56.242445 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:06:56.242602 (+ 157us) client_negotiation.cc:769] Sending connection context
0927 23:06:56.242754 (+ 152us) client_negotiation.cc:240] Negotiation successful
0927 23:06:56.243068 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136}
I20260927 23:06:56.250856 17593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c: Bootstrap starting.
I20260927 23:06:56.254767 17593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:56.256146 17593 log.cc:824] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:56.260656 17593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c: No bootstrap required, opened a new log
I20260927 23:06:56.271032 17593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.271400 17593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:56.271509 17593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02412a9de0b74aadabf856f7fc27ba4c, State: Initialized, Role: FOLLOWER
I20260927 23:06:56.272109 17593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.272998 17709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.273435 17709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:06:56.274188 17593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: configured and running, proceeding with master startup.
W20260927 23:06:56.277395 17473 tablet.cc:2369] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:06:56.285857 17720 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:06:56.285969 17720 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:06:56.289366 17526 master_runner.cc:263] No leader master found from master 02412a9de0b74aadabf856f7fc27ba4c
I20260927 23:06:56.289490 17526 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 23:06:56.348676 17709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:06:56.349016 17709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.350509 17709 leader_election.cc:288] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55f0c80e9afb477d9af773c8d635d759 (127.3.240.254:40233), 45adb01979b74d56892f7df2d9748f64 (127.3.240.252:39305)
I20260927 23:06:56.352723 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02412a9de0b74aadabf856f7fc27ba4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f0c80e9afb477d9af773c8d635d759" is_pre_election: true
I20260927 23:06:56.353471 17494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02412a9de0b74aadabf856f7fc27ba4c in term 0.
I20260927 23:06:56.354233 17544 leader_election.cc:302] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02412a9de0b74aadabf856f7fc27ba4c, 55f0c80e9afb477d9af773c8d635d759; no voters:
I20260927 23:06:56.354740 17709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:06:56.354923 17709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:06:56.355063 17709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:56.351456 17642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02412a9de0b74aadabf856f7fc27ba4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45adb01979b74d56892f7df2d9748f64" is_pre_election: true
I20260927 23:06:56.359369 17642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02412a9de0b74aadabf856f7fc27ba4c in term 0.
I20260927 23:06:56.360085 17709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.361133 17494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02412a9de0b74aadabf856f7fc27ba4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55f0c80e9afb477d9af773c8d635d759"
I20260927 23:06:56.361356 17494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:56.365893 17494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02412a9de0b74aadabf856f7fc27ba4c in term 1.
I20260927 23:06:56.367961 17544 leader_election.cc:302] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02412a9de0b74aadabf856f7fc27ba4c, 55f0c80e9afb477d9af773c8d635d759; no voters:
I20260927 23:06:56.369521 17642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02412a9de0b74aadabf856f7fc27ba4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45adb01979b74d56892f7df2d9748f64"
I20260927 23:06:56.369745 17642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:56.373252 17642 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02412a9de0b74aadabf856f7fc27ba4c in term 1.
I20260927 23:06:56.373828 17709 leader_election.cc:288] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [CANDIDATE]: Term 1 election: Requested vote from peers 55f0c80e9afb477d9af773c8d635d759 (127.3.240.254:40233), 45adb01979b74d56892f7df2d9748f64 (127.3.240.252:39305)
I20260927 23:06:56.374123 17709 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:06:56.374258 17709 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [term 1 LEADER]: Becoming Leader. State: Replica: 02412a9de0b74aadabf856f7fc27ba4c, State: Running, Role: LEADER
I20260927 23:06:56.375912 17709 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } }
I20260927 23:06:56.388640 17722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02412a9de0b74aadabf856f7fc27ba4c. Latest consensus state: current_term: 1 leader_uuid: "02412a9de0b74aadabf856f7fc27ba4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.393081 17722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:56.397071 17724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:06:56.399219 17724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:06:56.414672 17642 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [term 1 FOLLOWER]: Refusing update from remote peer 02412a9de0b74aadabf856f7fc27ba4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:06:56.418716 17722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:06:56.427012 17494 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [term 1 FOLLOWER]: Refusing update from remote peer 02412a9de0b74aadabf856f7fc27ba4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:06:56.427888 17722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:06:56.459966 17675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02412a9de0b74aadabf856f7fc27ba4c. Latest consensus state: current_term: 1 leader_uuid: "02412a9de0b74aadabf856f7fc27ba4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.460325 17675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:06:56.467737 17675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02412a9de0b74aadabf856f7fc27ba4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.467941 17675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:06:56.475873 17722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02412a9de0b74aadabf856f7fc27ba4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.476106 17722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:56.475524 17681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02412a9de0b74aadabf856f7fc27ba4c. Latest consensus state: current_term: 1 leader_uuid: "02412a9de0b74aadabf856f7fc27ba4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.476449 17681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:06:56.477532 17724 catalog_manager.cc:1347] Generated new cluster ID: a159adecfc9d46cfa6eda4ea9ba92e5b
I20260927 23:06:56.477635 17724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:06:56.491460 17732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02412a9de0b74aadabf856f7fc27ba4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.492010 17732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:56.493336 17681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02412a9de0b74aadabf856f7fc27ba4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55f0c80e9afb477d9af773c8d635d759" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40233 } } peers { permanent_uuid: "02412a9de0b74aadabf856f7fc27ba4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33311 } } peers { permanent_uuid: "45adb01979b74d56892f7df2d9748f64" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39305 } } }
I20260927 23:06:56.493584 17681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:06:56.498835 17724 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:06:56.500901 17724 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:06:56.540865 17724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02412a9de0b74aadabf856f7fc27ba4c: Generated new TSK 0
I20260927 23:06:56.542017 17724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:06:57.076637 17694 catalog_manager.cc:1259] Loaded cluster ID: a159adecfc9d46cfa6eda4ea9ba92e5b
I20260927 23:06:57.076756 17694 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: loading cluster ID for follower catalog manager: success
I20260927 23:06:57.079309 17694 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: acquiring CA information for follower catalog manager: success
I20260927 23:06:57.080935 17694 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55f0c80e9afb477d9af773c8d635d759: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 23:06:57.117722 17706 catalog_manager.cc:1259] Loaded cluster ID: a159adecfc9d46cfa6eda4ea9ba92e5b
I20260927 23:06:57.117834 17706 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: loading cluster ID for follower catalog manager: success
I20260927 23:06:57.120260 17706 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: acquiring CA information for follower catalog manager: success
I20260927 23:06:57.121966 17706 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45adb01979b74d56892f7df2d9748f64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 23:06:57.629655 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 17452
I20260927 23:06:57.630985 17514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:57.739236 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 17452
I20260927 23:06:57.759310 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 17526
I20260927 23:06:57.760457 17588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:57.866472 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 17526
I20260927 23:06:57.886138 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 17600
I20260927 23:06:57.887212 17662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:57.992290 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 17600
2026-09-27T23:06:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4666 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:06:58 dist-test-slave-lmfp krb5kdc[17745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:06:58 dist-test-slave-lmfp krb5kdc[17745](info): set up 2 sockets
Sep 27 23:06:58 dist-test-slave-lmfp krb5kdc[17745](info): commencing operation
krb5kdc: starting...
W20260927 23:06:59.051684 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:06:59 dist-test-slave-lmfp krb5kdc[17745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:06:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:06:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:06:59.273911 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42083
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:42083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:06:59.588946 17758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:06:59.589296 17758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:06:59.589445 17758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:06:59.596607 17758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:06:59.596745 17758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:06:59.596846 17758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:06:59.596940 17758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:06:59.597015 17758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:06:59.606594 17758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:42083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:06:59.607950 17758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:06:59.609249 17758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:06:59.620743 17766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:59.620705 17764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:06:59.620788 17763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:06:59.622010 17758 server_base.cc:1034] running on GCE node
I20260927 23:06:59.622781 17758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:06:59.624137 17758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:06:59.625370 17758 hybrid_clock.cc:648] HybridClock initialized: now 1790550419625321 us; error 61 us; skew 500 ppm
I20260927 23:06:59.625749 17758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:06:59 dist-test-slave-lmfp krb5kdc[17745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550419, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:06:59.628664 17758 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:06:59.630128 17758 webserver.cc:460] Webserver started at http://127.3.240.254:42403/ using document root <none> and password file <none>
I20260927 23:06:59.630785 17758 fs_manager.cc:359] Metadata directory not provided
I20260927 23:06:59.630909 17758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:06:59.631214 17758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:06:59.634162 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "f8c65d005dc040f3baac546f14f959d8"
format_stamp: "Formatted at 2026-09-27 23:06:59 on dist-test-slave-lmfp"
server_key: "566f7ec89ee386b4f2c366e41668a27f"
server_key_iv: "e4d8c73f59e4b808430f70a3e31e221b"
server_key_version: "encryptionkey@0"
I20260927 23:06:59.634943 17758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "f8c65d005dc040f3baac546f14f959d8"
format_stamp: "Formatted at 2026-09-27 23:06:59 on dist-test-slave-lmfp"
server_key: "566f7ec89ee386b4f2c366e41668a27f"
server_key_iv: "e4d8c73f59e4b808430f70a3e31e221b"
server_key_version: "encryptionkey@0"
I20260927 23:06:59.640048 17758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 23:06:59.643682 17773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:06:59.644990 17758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 23:06:59.645169 17758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "f8c65d005dc040f3baac546f14f959d8"
format_stamp: "Formatted at 2026-09-27 23:06:59 on dist-test-slave-lmfp"
server_key: "566f7ec89ee386b4f2c366e41668a27f"
server_key_iv: "e4d8c73f59e4b808430f70a3e31e221b"
server_key_version: "encryptionkey@0"
I20260927 23:06:59.645366 17758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:06:59.657850 17758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:06:59.662166 17758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:06:59.662592 17758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:06:59.681202 17758 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:42083
I20260927 23:06:59.681247 17824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:42083 every 8 connection(s)
I20260927 23:06:59.682685 17758 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:06:59.686424 17825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:06:59.692931 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 17758
I20260927 23:06:59.693218 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:06:59.694057 4035 external_mini_cluster.cc:1428] Setting key 7c4554e2b4c9ac9ed8e94cce3c428855
I20260927 23:06:59.697497 17825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8: Bootstrap starting.
I20260927 23:06:59.700559 17825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8: Neither blocks nor log segments found. Creating new log.
I20260927 23:06:59.701582 17825 log.cc:824] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8: Log is configured to *not* fsync() on all Append() calls
I20260927 23:06:59.704319 17825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8: No bootstrap required, opened a new log
Sep 27 23:06:59 dist-test-slave-lmfp krb5kdc[17745](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:06:59.712510 17825 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c65d005dc040f3baac546f14f959d8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42083 } }
I20260927 23:06:59.712855 17825 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:06:59.712988 17825 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c65d005dc040f3baac546f14f959d8, State: Initialized, Role: FOLLOWER
I20260927 23:06:59.713603 17825 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c65d005dc040f3baac546f14f959d8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42083 } }
I20260927 23:06:59.713824 17825 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:06:59.713923 17825 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:06:59.714124 17825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:06:59.716055 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:06:59.695934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.3.240.254:42083)
0927 23:06:59.696687 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0927 23:06:59.696698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:06:59.696738 (+ 40us) server_negotiation.cc:407] Connection header received
0927 23:06:59.698500 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:06:59.698522 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:06:59.698916 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:06:59.699390 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:06:59.700948 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:59.702314 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:59.703654 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:06:59.704318 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:06:59.704497 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:06:59.708798 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:06:59.708805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:06:59.708821 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:06:59.708853 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:06:59.711889 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:59.712610 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:59.712616 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:59.712622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:59.712789 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:06:59.713259 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:06:59.713264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:06:59.713281 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:06:59.713864 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:06:59.714162 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 23:06:59.714595 (+ 433us) server_negotiation.cc:299] Negotiation successful
0927 23:06:59.714977 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":359,"threads_started":1}
I20260927 23:06:59.716948 17825 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c65d005dc040f3baac546f14f959d8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42083 } }
I20260927 23:06:59.717501 17825 leader_election.cc:302] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8c65d005dc040f3baac546f14f959d8; no voters:
I20260927 23:06:59.718139 17825 leader_election.cc:288] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:06:59.718330 17830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:06:59.719947 17830 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [term 1 LEADER]: Becoming Leader. State: Replica: f8c65d005dc040f3baac546f14f959d8, State: Running, Role: LEADER
I20260927 23:06:59.720335 17825 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:06:59.720643 17830 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c65d005dc040f3baac546f14f959d8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42083 } }
I20260927 23:06:59.720980 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:06:59.723490 17831 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8c65d005dc040f3baac546f14f959d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c65d005dc040f3baac546f14f959d8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42083 } } }
I20260927 23:06:59.723907 17831 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:59.725240 17832 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c65d005dc040f3baac546f14f959d8. Latest consensus state: current_term: 1 leader_uuid: "f8c65d005dc040f3baac546f14f959d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c65d005dc040f3baac546f14f959d8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42083 } } }
I20260927 23:06:59.725586 17832 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8 [sys.catalog]: This master's current role is: LEADER
I20260927 23:06:59.728009 17836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:06:59.732524 17836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:06:59.741039 17836 catalog_manager.cc:1347] Generated new cluster ID: 66d17663218d4da185c1fd80955ac5d5
I20260927 23:06:59.741173 17836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:06:59.763891 17836 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:06:59.765352 17836 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:06:59.779270 17836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8c65d005dc040f3baac546f14f959d8: Generated new TSK 0
I20260927 23:06:59.780102 17836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:06:59.834925 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 17758
I20260927 23:06:59.836232 17820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:06:59.932094 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 17758
2026-09-27T23:06:59Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1938 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:07:00 dist-test-slave-lmfp krb5kdc[17853](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:07:00 dist-test-slave-lmfp krb5kdc[17853](info): set up 2 sockets
Sep 27 23:07:00 dist-test-slave-lmfp krb5kdc[17853](info): commencing operation
krb5kdc: starting...
W20260927 23:07:01.003857 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:07:01 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:07:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:07:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:07:01.216399 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46819
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:01.541049 17866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:01.541329 17866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:01.541512 17866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:01.548580 17866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:07:01.548686 17866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:01.548836 17866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:01.548929 17866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:07:01.549008 17866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:07:01.558519 17866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:01.559825 17866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:01.561151 17866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:01.573035 17872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:01.573113 17871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:01.573089 17874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:01.574108 17866 server_base.cc:1034] running on GCE node
I20260927 23:07:01.574795 17866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:01.576134 17866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:01.577327 17866 hybrid_clock.cc:648] HybridClock initialized: now 1790550421577293 us; error 26 us; skew 500 ppm
I20260927 23:07:01.577607 17866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:01 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:01.583379 17866 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:07:01.584728 17866 webserver.cc:460] Webserver started at http://127.3.240.254:45793/ using document root <none> and password file <none>
I20260927 23:07:01.585372 17866 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:01.585496 17866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:01.585781 17866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:07:01.588687 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "5952c24970de4118b1df3704a7eb4f91"
format_stamp: "Formatted at 2026-09-27 23:07:01 on dist-test-slave-lmfp"
server_key: "399a2a09d4ca3dd4b1d155c21e9bcf18"
server_key_iv: "7c711b08280303d8907e8cbc8b4175db"
server_key_version: "encryptionkey@0"
I20260927 23:07:01.589435 17866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "5952c24970de4118b1df3704a7eb4f91"
format_stamp: "Formatted at 2026-09-27 23:07:01 on dist-test-slave-lmfp"
server_key: "399a2a09d4ca3dd4b1d155c21e9bcf18"
server_key_iv: "7c711b08280303d8907e8cbc8b4175db"
server_key_version: "encryptionkey@0"
I20260927 23:07:01.594511 17866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 23:07:01.597759 17881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:01.599076 17866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:07:01.599248 17866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "5952c24970de4118b1df3704a7eb4f91"
format_stamp: "Formatted at 2026-09-27 23:07:01 on dist-test-slave-lmfp"
server_key: "399a2a09d4ca3dd4b1d155c21e9bcf18"
server_key_iv: "7c711b08280303d8907e8cbc8b4175db"
server_key_version: "encryptionkey@0"
I20260927 23:07:01.599438 17866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:01.612641 17866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:01.616240 17866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:01.616494 17866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:01.633706 17866 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46819
I20260927 23:07:01.633744 17932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46819 every 8 connection(s)
I20260927 23:07:01.634958 17866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:07:01.636211 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 17866
I20260927 23:07:01.636415 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:07:01.637226 4035 external_mini_cluster.cc:1428] Setting key 13b00023fee017fe9bfb7fe834b1e532
I20260927 23:07:01.639451 17933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 23:07:01 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:01.652912 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Bootstrap starting.
I20260927 23:07:01.656387 17933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:01.657691 17933 log.cc:824] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:01.660607 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:01.639262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.3.240.254:46819)
0927 23:07:01.640120 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:01.640134 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:01.640185 (+ 51us) server_negotiation.cc:407] Connection header received
0927 23:07:01.642288 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:01.642314 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:01.642796 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:01.643226 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:01.644444 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:01.646426 (+ 1982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:01.647768 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:01.648741 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:01.648941 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:01.653623 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:01.653631 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:01.653644 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:01.653678 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:01.656600 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:01.657435 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:01.657441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:01.657446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:01.657606 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:01.657993 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:01.657996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:01.657998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:01.658696 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:01.659071 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:01.659340 (+ 269us) server_negotiation.cc:299] Negotiation successful
0927 23:07:01.659677 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":370,"threads_started":1}
I20260927 23:07:01.660754 17933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: No bootstrap required, opened a new log
I20260927 23:07:01.667562 17933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:01.667879 17933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:01.667986 17933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5952c24970de4118b1df3704a7eb4f91, State: Initialized, Role: FOLLOWER
I20260927 23:07:01.668503 17933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:01.668733 17933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:07:01.668895 17933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:07:01.669064 17933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:01.671155 17933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:01.671631 17933 leader_election.cc:302] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5952c24970de4118b1df3704a7eb4f91; no voters:
I20260927 23:07:01.672137 17933 leader_election.cc:288] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:07:01.672251 17938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:07:01.674067 17938 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 1 LEADER]: Becoming Leader. State: Replica: 5952c24970de4118b1df3704a7eb4f91, State: Running, Role: LEADER
I20260927 23:07:01.674604 17938 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:01.675153 17933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:07:01.677338 17940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5952c24970de4118b1df3704a7eb4f91. Latest consensus state: current_term: 1 leader_uuid: "5952c24970de4118b1df3704a7eb4f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } } }
I20260927 23:07:01.677305 17939 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5952c24970de4118b1df3704a7eb4f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } } }
I20260927 23:07:01.677747 17940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:01.677778 17939 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:01.678956 17945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:07:01.684293 17945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:07:01.692822 17945 catalog_manager.cc:1347] Generated new cluster ID: 2f215d23fd5a4f2dab067bcf3e01e41a
I20260927 23:07:01.692991 17945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:07:01.706781 17945 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:07:01.708026 17945 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:07:01.715034 17945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Generated new TSK 0
I20260927 23:07:01.715802 17945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:07:01.776038 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46819
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:02.119719 17959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:02.119987 17959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:02.120082 17959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:02.127182 17959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:02.127293 17959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:02.127432 17959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:07:02.137030 17959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:02.138283 17959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:02.139550 17959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:02.152137 17964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:02.152237 17965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:02.152297 17967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:02.153083 17959 server_base.cc:1034] running on GCE node
I20260927 23:07:02.153676 17959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:02.154633 17959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:02.155848 17959 hybrid_clock.cc:648] HybridClock initialized: now 1790550422155793 us; error 52 us; skew 500 ppm
I20260927 23:07:02.156153 17959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:02 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550422, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:02.159350 17959 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:07:02.160913 17959 webserver.cc:460] Webserver started at http://127.3.240.193:34187/ using document root <none> and password file <none>
I20260927 23:07:02.161573 17959 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:02.161705 17959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:02.162075 17959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:07:02.165047 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "fe2c285e7e9946e2b7ba2f5053698346"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "eeef7439452a04928bcbfe6c151218d0"
server_key_iv: "4760cb13d542780bc9a663cd92d85cc7"
server_key_version: "encryptionkey@0"
I20260927 23:07:02.165727 17959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "fe2c285e7e9946e2b7ba2f5053698346"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "eeef7439452a04928bcbfe6c151218d0"
server_key_iv: "4760cb13d542780bc9a663cd92d85cc7"
server_key_version: "encryptionkey@0"
I20260927 23:07:02.170485 17959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260927 23:07:02.173774 17974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:02.175074 17959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:07:02.175227 17959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "fe2c285e7e9946e2b7ba2f5053698346"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "eeef7439452a04928bcbfe6c151218d0"
server_key_iv: "4760cb13d542780bc9a663cd92d85cc7"
server_key_version: "encryptionkey@0"
I20260927 23:07:02.175417 17959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:02.189275 17959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:02.193372 17959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:02.193663 17959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:02.194609 17959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:02.196130 17959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:02.196235 17959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:02.196385 17959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:02.196458 17959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:02.228493 17959 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:43007
I20260927 23:07:02.228540 18086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:43007 every 8 connection(s)
I20260927 23:07:02.229744 17959 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:07:02.239971 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 17959
I20260927 23:07:02.240195 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:07:02.240782 4035 external_mini_cluster.cc:1428] Setting key c4c55e136f002eb8a1e1d4463f3832fa
Sep 27 23:07:02 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550422, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:02.249235 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:02.232200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:48427 (local address 127.3.240.254:46819)
0927 23:07:02.232635 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:02.232640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:02.233693 (+ 1053us) server_negotiation.cc:407] Connection header received
0927 23:07:02.235578 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:02.235583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:02.235662 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:02.235790 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:02.237203 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:02.237922 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:02.239565 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:02.239952 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:02.240095 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:02.243766 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:02.243771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:02.243774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:02.243804 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:02.246250 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:02.246954 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:02.246958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:02.246960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:02.247131 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:02.247570 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:02.247575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:02.247578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:02.247832 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:02.247996 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:02.248640 (+ 644us) server_negotiation.cc:299] Negotiation successful
0927 23:07:02.248878 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":194,"threads_started":1}
I20260927 23:07:02.250197 18090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:02.232785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:48427)
0927 23:07:02.233531 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:02.233573 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:02.235323 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:02.236005 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:02.236013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:02.236505 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:02.237037 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:02.237055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:02.238080 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:02.238084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:02.239414 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:02.239426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:02.240175 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:02.240183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:02.240453 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:02.241131 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:02.241160 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:02.243551 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:02.246403 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:02.246411 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:02.246427 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:02.246828 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:02.247235 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:02.247240 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:02.247243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:02.247449 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:02.247952 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:02.247960 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:02.248291 (+ 331us) client_negotiation.cc:769] Sending connection context
0927 23:07:02.248534 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 23:07:02.249051 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":297,"threads_started":1}
I20260927 23:07:02.251757 18087 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46819
I20260927 23:07:02.252045 18087 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:02.252729 18087 heartbeater.cc:511] Master 127.3.240.254:46819 requested a full tablet report, sending...
I20260927 23:07:02.254916 17898 ts_manager.cc:194] Registered new tserver with Master: fe2c285e7e9946e2b7ba2f5053698346 (127.3.240.193:43007)
I20260927 23:07:02.257059 17898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:48427
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:07:02.293649 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46819
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:02.602777 18093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:02.603181 18093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:02.603351 18093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:02.610150 18093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:02.610303 18093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:02.610472 18093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:07:02.619858 18093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:02.621086 18093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:02.622380 18093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:02.634961 18099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:02.635429 18098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:02.635046 18101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:02.636150 18093 server_base.cc:1034] running on GCE node
I20260927 23:07:02.636709 18093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:02.637602 18093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:02.638824 18093 hybrid_clock.cc:648] HybridClock initialized: now 1790550422638781 us; error 45 us; skew 500 ppm
I20260927 23:07:02.639161 18093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:02 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550422, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:02.642141 18093 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:07:02.643661 18093 webserver.cc:460] Webserver started at http://127.3.240.194:39051/ using document root <none> and password file <none>
I20260927 23:07:02.644328 18093 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:02.644444 18093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:02.644727 18093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:07:02.647882 18093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "b0b96907e0944127b05425e5559c1444"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "73d5b67bf19d319a5ba54f2c437c0678"
server_key_iv: "bbf0f1702811fb79159cfdbdb64395aa"
server_key_version: "encryptionkey@0"
I20260927 23:07:02.648602 18093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "b0b96907e0944127b05425e5559c1444"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "73d5b67bf19d319a5ba54f2c437c0678"
server_key_iv: "bbf0f1702811fb79159cfdbdb64395aa"
server_key_version: "encryptionkey@0"
I20260927 23:07:02.653473 18093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 23:07:02.656687 18108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:02.657850 18093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:07:02.658011 18093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "b0b96907e0944127b05425e5559c1444"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "73d5b67bf19d319a5ba54f2c437c0678"
server_key_iv: "bbf0f1702811fb79159cfdbdb64395aa"
server_key_version: "encryptionkey@0"
I20260927 23:07:02.658187 18093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:02.674569 18093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:02.678289 18093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:02.678623 18093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:02.679607 18093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:02.680898 18093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:02.680966 18093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:02.681100 18093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:02.681172 18093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:02.712646 18093 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:46443
I20260927 23:07:02.712682 18220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:46443 every 8 connection(s)
I20260927 23:07:02.713943 18093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:07:02.724313 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 18093
I20260927 23:07:02.724500 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:07:02.725067 4035 external_mini_cluster.cc:1428] Setting key 59ff9c51dbb71bb0718f650669562c52
Sep 27 23:07:02 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550422, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:02.733736 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:02.716350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:55317 (local address 127.3.240.254:46819)
0927 23:07:02.716541 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:02.716547 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:02.717655 (+ 1108us) server_negotiation.cc:407] Connection header received
0927 23:07:02.719340 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:02.719344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:02.719407 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:02.719482 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:02.720862 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:02.721521 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:02.722912 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:02.723379 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:02.723467 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:02.727217 (+ 3750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:02.727223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:02.727227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:02.727274 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:02.730559 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:02.731313 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:02.731321 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:02.731327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:02.731474 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:02.731872 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:02.731879 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:02.731885 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:02.732258 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:02.732420 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:02.733054 (+ 634us) server_negotiation.cc:299] Negotiation successful
0927 23:07:02.733308 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 23:07:02.734297 18223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:02.716858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:55317)
0927 23:07:02.717502 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:02.717543 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:02.719091 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:02.719671 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:02.719678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:02.720165 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:02.720726 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:02.720741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:02.721639 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:02.721642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:02.722798 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:02.722808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:02.723546 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:02.723551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:02.723729 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:02.724377 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:02.724417 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:02.726987 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:02.730682 (+ 3695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:02.730688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:02.730700 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:02.731193 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:02.731583 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:02.731586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:02.731589 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:02.731759 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:02.732393 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:02.732398 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:02.732703 (+ 305us) client_negotiation.cc:769] Sending connection context
0927 23:07:02.732921 (+ 218us) client_negotiation.cc:240] Negotiation successful
0927 23:07:02.733265 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":245,"threads_started":1}
I20260927 23:07:02.735829 18221 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46819
I20260927 23:07:02.736140 18221 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:02.736883 18221 heartbeater.cc:511] Master 127.3.240.254:46819 requested a full tablet report, sending...
I20260927 23:07:02.738219 17898 ts_manager.cc:194] Registered new tserver with Master: b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443)
I20260927 23:07:02.739074 17898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:55317
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:07:02.773945 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46819
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:03.080770 18226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:03.081044 18226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:03.081213 18226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:03.088119 18226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:03.088270 18226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:03.088438 18226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:07:03.098114 18226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:03.099512 18226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:03.100831 18226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:03.115923 18232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:03.116153 18234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:03.116716 18231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:03.119941 18226 server_base.cc:1034] running on GCE node
I20260927 23:07:03.120728 18226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:03.121968 18226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:03.123185 18226 hybrid_clock.cc:648] HybridClock initialized: now 1790550423123135 us; error 48 us; skew 500 ppm
I20260927 23:07:03.123517 18226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:03 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550423, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:03.126479 18226 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:07:03.128029 18226 webserver.cc:460] Webserver started at http://127.3.240.195:37647/ using document root <none> and password file <none>
I20260927 23:07:03.128688 18226 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:03.128829 18226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:03.129109 18226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:07:03.132053 18226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "2ce1276eca724fafba55c45a82ac0d7a"
format_stamp: "Formatted at 2026-09-27 23:07:03 on dist-test-slave-lmfp"
server_key: "d19413a604a9104e32a4399eeb1b1721"
server_key_iv: "d05f1cf3488f9784079abd84757c4393"
server_key_version: "encryptionkey@0"
I20260927 23:07:03.132795 18226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "2ce1276eca724fafba55c45a82ac0d7a"
format_stamp: "Formatted at 2026-09-27 23:07:03 on dist-test-slave-lmfp"
server_key: "d19413a604a9104e32a4399eeb1b1721"
server_key_iv: "d05f1cf3488f9784079abd84757c4393"
server_key_version: "encryptionkey@0"
I20260927 23:07:03.137781 18226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260927 23:07:03.141040 18241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:03.142366 18226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:07:03.142553 18226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "2ce1276eca724fafba55c45a82ac0d7a"
format_stamp: "Formatted at 2026-09-27 23:07:03 on dist-test-slave-lmfp"
server_key: "d19413a604a9104e32a4399eeb1b1721"
server_key_iv: "d05f1cf3488f9784079abd84757c4393"
server_key_version: "encryptionkey@0"
I20260927 23:07:03.142738 18226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:03.163908 18226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:03.167717 18226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:03.167975 18226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:03.168952 18226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:03.170226 18226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:03.170327 18226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:03.170461 18226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:03.170538 18226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:03.205201 18226 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33987
I20260927 23:07:03.205281 18353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33987 every 8 connection(s)
I20260927 23:07:03.206415 18226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:07:03.214458 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 18226
I20260927 23:07:03.214632 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:07:03.215137 4035 external_mini_cluster.cc:1428] Setting key fbbe398c2e833a64188e13b4c1313d0b
Sep 27 23:07:03 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550423, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:03.225883 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.208940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:57535 (local address 127.3.240.254:46819)
0927 23:07:03.209143 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.209149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.210270 (+ 1121us) server_negotiation.cc:407] Connection header received
0927 23:07:03.212013 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.212017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.212122 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.212211 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:03.213667 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.214629 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.216639 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.217155 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.217296 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.220800 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:03.220804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:03.220808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:03.220839 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:03.223431 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.224059 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.224063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.224065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.224178 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.224523 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.224526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.224528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.224765 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:03.224865 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.225492 (+ 627us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.225637 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 23:07:03.226835 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.209422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:57535)
0927 23:07:03.210093 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:03.210145 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:03.211748 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:03.212447 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:03.212455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:03.212956 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:03.213524 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.213539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.214823 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.214827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.216501 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.216514 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.217373 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.217378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.217582 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.218411 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:03.218447 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:03.220622 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:03.223539 (+ 2917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:03.223546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:03.223558 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:03.223960 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:03.224268 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:03.224272 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:03.224274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:03.224434 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:03.224858 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:03.224863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:03.225149 (+ 286us) client_negotiation.cc:769] Sending connection context
0927 23:07:03.225388 (+ 239us) client_negotiation.cc:240] Negotiation successful
0927 23:07:03.225737 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":264,"threads_started":1}
I20260927 23:07:03.228219 18354 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46819
I20260927 23:07:03.228503 18354 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:03.229327 18354 heartbeater.cc:511] Master 127.3.240.254:46819 requested a full tablet report, sending...
I20260927 23:07:03.230844 17898 ts_manager.cc:194] Registered new tserver with Master: 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987)
I20260927 23:07:03.231742 17898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:57535
I20260927 23:07:03.243144 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:07:03.259204 18089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.247293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.3.240.254:46819)
0927 23:07:03.247460 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.247468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.247833 (+ 365us) server_negotiation.cc:407] Connection header received
0927 23:07:03.248010 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.248015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.248078 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.248162 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:03.249467 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.250151 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.251546 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.251945 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.252072 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.254329 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:03.254335 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:03.254339 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:03.254377 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:03.256675 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.257342 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.257347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.257351 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.257472 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.257814 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.257820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.257823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.258122 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:03.258360 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.258655 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.258862 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260927 23:07:03.259958 18087 heartbeater.cc:503] Master 127.3.240.254:46819 was elected leader, sending a full tablet report...
I20260927 23:07:03.262954 17898 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:07:03.269733 17898 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:07:03.294489 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.283340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.3.240.193:43007)
0927 23:07:03.286233 (+ 2893us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.286240 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.286263 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:07:03.286337 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.286342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.286521 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.286701 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:03.287443 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.288865 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.291860 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.293190 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.293352 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.293591 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.294029 (+ 438us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.294186 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2711,"thread_start_us":2448,"threads_started":1}
I20260927 23:07:03.294534 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.282551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.3.240.195:33987)
0927 23:07:03.287510 (+ 4959us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.287517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.287547 (+ 30us) server_negotiation.cc:407] Connection header received
0927 23:07:03.287638 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.287644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.287857 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.288032 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:03.288622 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.290001 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.291913 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.293242 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.293789 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.294000 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.294066 (+ 66us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.294208 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4782,"thread_start_us":4623,"threads_started":1}
I20260927 23:07:03.294552 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.282227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33987 (local address 127.0.0.1:47636)
0927 23:07:03.283531 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:03.283588 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:03.283800 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:03.288042 (+ 4242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:03.288048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:03.288073 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:03.288422 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.288434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.290125 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.290130 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.291719 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.291731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.293485 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.293491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.293711 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.293824 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 23:07:03.293892 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 23:07:03.294079 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":80,"thread_start_us":3068,"threads_started":1}
I20260927 23:07:03.295428 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.284593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40544 (local address 127.3.240.194:46443)
0927 23:07:03.285285 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.285297 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.285931 (+ 634us) server_negotiation.cc:407] Connection header received
0927 23:07:03.286059 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.286071 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.286539 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.286771 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:03.287666 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.289075 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.291824 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.293242 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.293390 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.293577 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.294946 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.295145 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":344,"threads_started":1}
I20260927 23:07:03.294593 18366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.283918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46443 (local address 127.0.0.1:40544)
0927 23:07:03.285640 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:03.285661 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:03.285810 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:03.287073 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:03.287080 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:03.287114 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:03.287513 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.287523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.289236 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.289241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.291663 (+ 2422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.291680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.293721 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.293726 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.293907 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.293934 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 23:07:03.294027 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 23:07:03.294142 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1524,"mutex_wait_us":10,"thread_start_us":893,"threads_started":1}
I20260927 23:07:03.295934 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.282618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:55434)
0927 23:07:03.284530 (+ 1912us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:03.284552 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:03.284677 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:03.286706 (+ 2029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:03.286713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:03.286739 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:03.287191 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.287210 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.289027 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.289034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.291663 (+ 2629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.291680 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.293401 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.293407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.293604 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.293719 (+ 115us) client_negotiation.cc:769] Sending connection context
0927 23:07:03.293862 (+ 143us) client_negotiation.cc:240] Negotiation successful
0927 23:07:03.295077 (+ 1215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1751,"mutex_wait_us":1155,"thread_start_us":412,"threads_started":1}
I20260927 23:07:03.298519 18156 tablet_service.cc:1467] Processing CreateTablet for tablet fe5defe191d8462f8f0592d65992f033 (DEFAULT_TABLE table=test-table [id=fc5321b670094994af7432a48cc8ff4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:03.298466 18289 tablet_service.cc:1467] Processing CreateTablet for tablet fe5defe191d8462f8f0592d65992f033 (DEFAULT_TABLE table=test-table [id=fc5321b670094994af7432a48cc8ff4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:03.299639 18156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5defe191d8462f8f0592d65992f033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:03.299858 18289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5defe191d8462f8f0592d65992f033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:03.299566 18022 tablet_service.cc:1467] Processing CreateTablet for tablet fe5defe191d8462f8f0592d65992f033 (DEFAULT_TABLE table=test-table [id=fc5321b670094994af7432a48cc8ff4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:03.300769 18022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe5defe191d8462f8f0592d65992f033. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:03.311098 18372 tablet_bootstrap.cc:492] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Bootstrap starting.
I20260927 23:07:03.314437 18374 tablet_bootstrap.cc:492] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Bootstrap starting.
I20260927 23:07:03.314543 18372 tablet_bootstrap.cc:654] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:03.315845 18372 log.cc:824] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:03.316098 18373 tablet_bootstrap.cc:492] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Bootstrap starting.
I20260927 23:07:03.318153 18374 tablet_bootstrap.cc:654] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:03.318745 18373 tablet_bootstrap.cc:654] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:03.319455 18374 log.cc:824] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:03.319939 18373 log.cc:824] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:03.321858 18372 tablet_bootstrap.cc:492] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: No bootstrap required, opened a new log
I20260927 23:07:03.322189 18372 ts_tablet_manager.cc:1397] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260927 23:07:03.325141 18373 tablet_bootstrap.cc:492] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: No bootstrap required, opened a new log
I20260927 23:07:03.325376 18373 ts_tablet_manager.cc:1397] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 23:07:03.325970 18374 tablet_bootstrap.cc:492] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: No bootstrap required, opened a new log
I20260927 23:07:03.326205 18374 ts_tablet_manager.cc:1397] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260927 23:07:03.331995 18372 raft_consensus.cc:348] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
I20260927 23:07:03.332425 18372 raft_consensus.cc:374] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:03.332222 18373 raft_consensus.cc:348] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
I20260927 23:07:03.332571 18372 raft_consensus.cc:729] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b96907e0944127b05425e5559c1444, State: Initialized, Role: FOLLOWER
I20260927 23:07:03.332592 18373 raft_consensus.cc:374] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:03.332717 18373 raft_consensus.cc:729] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce1276eca724fafba55c45a82ac0d7a, State: Initialized, Role: FOLLOWER
I20260927 23:07:03.333287 18373 consensus_queue.cc:260] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
I20260927 23:07:03.333289 18372 consensus_queue.cc:260] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
I20260927 23:07:03.334306 18354 heartbeater.cc:503] Master 127.3.240.254:46819 was elected leader, sending a full tablet report...
I20260927 23:07:03.334666 18221 heartbeater.cc:503] Master 127.3.240.254:46819 was elected leader, sending a full tablet report...
I20260927 23:07:03.334983 18373 ts_tablet_manager.cc:1428] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Time spent starting tablet: real 0.009s user 0.004s sys 0.006s
I20260927 23:07:03.335846 18374 raft_consensus.cc:348] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
I20260927 23:07:03.336316 18374 raft_consensus.cc:374] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:03.336448 18374 raft_consensus.cc:729] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2c285e7e9946e2b7ba2f5053698346, State: Initialized, Role: FOLLOWER
I20260927 23:07:03.336527 18372 ts_tablet_manager.cc:1428] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260927 23:07:03.337090 18374 consensus_queue.cc:260] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
I20260927 23:07:03.338567 18374 ts_tablet_manager.cc:1428] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260927 23:07:03.434973 18378 raft_consensus.cc:482] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:03.435320 18378 raft_consensus.cc:504] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
I20260927 23:07:03.436990 18378 leader_election.cc:288] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe2c285e7e9946e2b7ba2f5053698346 (127.3.240.193:43007), b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443)
I20260927 23:07:03.443470 18356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.437342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.3.240.195:49619)
0927 23:07:03.437529 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:03.437547 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:03.437671 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:03.438179 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:03.438184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:03.438219 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:03.438429 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.438444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.439686 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.439689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.441408 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.441423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.442834 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.442838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.442965 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.443040 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 23:07:03.443129 (+ 89us) client_negotiation.cc:240] Negotiation successful
0927 23:07:03.443221 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 23:07:03.443538 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.437457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:49619 (local address 127.3.240.193:43007)
0927 23:07:03.437621 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.437626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.437644 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:07:03.437811 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.437816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.437881 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.437989 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:03.438582 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.439568 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.441544 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.442693 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.442867 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.442944 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.443159 (+ 215us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.443287 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 23:07:03.444207 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.438047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:43941 (local address 127.3.240.194:46443)
0927 23:07:03.438232 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.438249 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.438461 (+ 212us) server_negotiation.cc:407] Connection header received
0927 23:07:03.438630 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.438637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.438707 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.438784 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:03.439261 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.440012 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.442085 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.443322 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.443467 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.443533 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.443854 (+ 321us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.443986 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:07:03.444360 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.437753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46443 (local address 127.3.240.195:43941)
0927 23:07:03.438367 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:03.438386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:03.438497 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:03.438810 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:03.438813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:03.438831 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:03.439124 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.439132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.440137 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.440141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.441952 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:03.441964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.443517 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.443526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:03.443704 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.443734 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 23:07:03.443834 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 23:07:03.443944 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":2,"thread_start_us":293,"threads_started":1}
I20260927 23:07:03.444945 18042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5defe191d8462f8f0592d65992f033" candidate_uuid: "2ce1276eca724fafba55c45a82ac0d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2c285e7e9946e2b7ba2f5053698346" is_pre_election: true
I20260927 23:07:03.445307 18176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5defe191d8462f8f0592d65992f033" candidate_uuid: "2ce1276eca724fafba55c45a82ac0d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b96907e0944127b05425e5559c1444" is_pre_election: true
I20260927 23:07:03.445441 18042 raft_consensus.cc:2454] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce1276eca724fafba55c45a82ac0d7a in term 0.
I20260927 23:07:03.445782 18176 raft_consensus.cc:2454] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce1276eca724fafba55c45a82ac0d7a in term 0.
I20260927 23:07:03.446084 18244 leader_election.cc:302] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce1276eca724fafba55c45a82ac0d7a, fe2c285e7e9946e2b7ba2f5053698346; no voters:
I20260927 23:07:03.446429 18378 raft_consensus.cc:2781] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:07:03.446565 18378 raft_consensus.cc:482] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:07:03.446671 18378 raft_consensus.cc:3037] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:03.448760 18378 raft_consensus.cc:504] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
I20260927 23:07:03.449241 18378 leader_election.cc:288] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [CANDIDATE]: Term 1 election: Requested vote from peers fe2c285e7e9946e2b7ba2f5053698346 (127.3.240.193:43007), b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443)
I20260927 23:07:03.449620 18042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5defe191d8462f8f0592d65992f033" candidate_uuid: "2ce1276eca724fafba55c45a82ac0d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2c285e7e9946e2b7ba2f5053698346"
I20260927 23:07:03.449679 18176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe5defe191d8462f8f0592d65992f033" candidate_uuid: "2ce1276eca724fafba55c45a82ac0d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b96907e0944127b05425e5559c1444"
I20260927 23:07:03.449785 18042 raft_consensus.cc:3037] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:03.449853 18176 raft_consensus.cc:3037] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:03.451817 18042 raft_consensus.cc:2454] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce1276eca724fafba55c45a82ac0d7a in term 1.
I20260927 23:07:03.451948 18176 raft_consensus.cc:2454] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce1276eca724fafba55c45a82ac0d7a in term 1.
I20260927 23:07:03.452170 18244 leader_election.cc:302] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce1276eca724fafba55c45a82ac0d7a, fe2c285e7e9946e2b7ba2f5053698346; no voters:
I20260927 23:07:03.452435 18378 raft_consensus.cc:2781] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:07:03.452844 18378 raft_consensus.cc:686] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 LEADER]: Becoming Leader. State: Replica: 2ce1276eca724fafba55c45a82ac0d7a, State: Running, Role: LEADER
I20260927 23:07:03.453372 18378 consensus_queue.cc:237] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } }
W20260927 23:07:03.457891 18355 tablet.cc:2369] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:07:03.457489 17895 catalog_manager.cc:5495] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195). New cstate: current_term: 1 leader_uuid: "2ce1276eca724fafba55c45a82ac0d7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } health_report { overall_health: UNKNOWN } } }
W20260927 23:07:03.465813 18222 tablet.cc:2369] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:07:03.481884 18088 tablet.cc:2369] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:07:03.489459 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.484986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.3.240.195:33987)
0927 23:07:03.485164 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.485168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.485183 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:07:03.485266 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.485269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.485345 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.485458 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:07:03.485750 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.486398 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.487730 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.488090 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.488175 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.488589 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:07:03.489038 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:07:03.489134 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.489214 (+ 80us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.489282 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:07:03.497179 18176 raft_consensus.cc:1261] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 1 FOLLOWER]: Refusing update from remote peer 2ce1276eca724fafba55c45a82ac0d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:07:03.497179 18042 raft_consensus.cc:1261] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Refusing update from remote peer 2ce1276eca724fafba55c45a82ac0d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:07:03.498065 18382 consensus_queue.cc:1035] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:07:03.498405 18378 consensus_queue.cc:1035] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:07:03.522610 17896 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:07:03.522877 17896 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:07:03.526103 17896 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Sending DeleteTablet for 3 replicas of tablet fe5defe191d8462f8f0592d65992f033
I20260927 23:07:03.527173 18289 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5defe191d8462f8f0592d65992f033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:03 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:47636
I20260927 23:07:03.527318 18022 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5defe191d8462f8f0592d65992f033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:03 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:55434
I20260927 23:07:03.527488 18156 tablet_service.cc:1514] Processing DeleteTablet for tablet fe5defe191d8462f8f0592d65992f033 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:03 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:40544
I20260927 23:07:03.528031 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 17959
I20260927 23:07:03.538087 18396 tablet_replica.cc:330] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: stopping tablet replica
I20260927 23:07:03.539613 18396 raft_consensus.cc:2229] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:07:03.539799 18395 tablet_replica.cc:330] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: stopping tablet replica
I20260927 23:07:03.540339 18396 raft_consensus.cc:2258] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:03.540784 18395 raft_consensus.cc:2229] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:07:03.540877 18397 tablet_replica.cc:330] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: stopping tablet replica
I20260927 23:07:03.541590 18395 raft_consensus.cc:2258] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:03.541604 18397 raft_consensus.cc:2229] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:07:03.542238 18397 raft_consensus.cc:2258] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:03.543780 18395 ts_tablet_manager.cc:1905] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:03.543773 18396 ts_tablet_manager.cc:1905] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:03.546345 18397 ts_tablet_manager.cc:1905] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:03.550441 18395 ts_tablet_manager.cc:1918] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:03.550650 18395 log.cc:1196] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/fe5defe191d8462f8f0592d65992f033
I20260927 23:07:03.551280 18395 ts_tablet_manager.cc:1939] T fe5defe191d8462f8f0592d65992f033 P 2ce1276eca724fafba55c45a82ac0d7a: Deleting consensus metadata
I20260927 23:07:03.552914 18396 ts_tablet_manager.cc:1918] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:03.553052 17885 catalog_manager.cc:4841] TS 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987): tablet fe5defe191d8462f8f0592d65992f033 (table test-table [id=fc5321b670094994af7432a48cc8ff4d]) successfully deleted
I20260927 23:07:03.553151 18396 log.cc:1196] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/fe5defe191d8462f8f0592d65992f033
I20260927 23:07:03.553673 18396 ts_tablet_manager.cc:1939] T fe5defe191d8462f8f0592d65992f033 P fe2c285e7e9946e2b7ba2f5053698346: Deleting consensus metadata
I20260927 23:07:03.555557 17884 catalog_manager.cc:4841] TS fe2c285e7e9946e2b7ba2f5053698346 (127.3.240.193:43007): tablet fe5defe191d8462f8f0592d65992f033 (table test-table [id=fc5321b670094994af7432a48cc8ff4d]) successfully deleted
I20260927 23:07:03.555974 18397 ts_tablet_manager.cc:1918] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:03.556121 18397 log.cc:1196] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/fe5defe191d8462f8f0592d65992f033
I20260927 23:07:03.556629 18397 ts_tablet_manager.cc:1939] T fe5defe191d8462f8f0592d65992f033 P b0b96907e0944127b05425e5559c1444: Deleting consensus metadata
Sep 27 23:07:03 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:07:03.558277 17884 catalog_manager.cc:4841] TS b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443): tablet fe5defe191d8462f8f0592d65992f033 (table test-table [id=fc5321b670094994af7432a48cc8ff4d]) successfully deleted
I20260927 23:07:03.563419 18369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.529005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55450 (local address 127.3.240.193:43007)
0927 23:07:03.529158 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.529163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.542678 (+ 13515us) server_negotiation.cc:407] Connection header received
0927 23:07:03.543151 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.543157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.543254 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.543329 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:03.545307 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.546412 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.548358 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.548977 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.549067 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.557795 (+ 8728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:03.557806 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:03.557819 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:03.557851 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:03.560742 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.561415 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.561422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.561424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.561560 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.561936 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.561941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.561943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.562313 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:03.562610 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.562905 (+ 295us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.563124 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 23:07:03.564013 18082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:03.672134 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 17959
I20260927 23:07:03.689913 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 18093
Sep 27 23:07:03 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:07:03.707301 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.690831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.3.240.194:46443)
0927 23:07:03.691046 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.691052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.691078 (+ 26us) server_negotiation.cc:407] Connection header received
0927 23:07:03.691238 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.691241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.691312 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.691413 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:03.693047 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.693780 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.695652 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.696020 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.696115 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.700822 (+ 4707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:03.700839 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:03.700859 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:03.700899 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:03.704602 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.705215 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.705225 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.705229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.705390 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.705728 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.705733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.705736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.706177 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:03.706464 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.706666 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.706875 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 23:07:03.708052 18216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:03.793574 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 18093
I20260927 23:07:03.810194 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 18226
Sep 27 23:07:03 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:07:03.827956 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:03.811013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47660 (local address 127.3.240.195:33987)
0927 23:07:03.811189 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:03.811193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:03.811212 (+ 19us) server_negotiation.cc:407] Connection header received
0927 23:07:03.811409 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:03.811412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:03.811480 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:03.811555 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:03.813335 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.814144 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.815966 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:03.816407 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:03.816509 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:03.821779 (+ 5270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:03.821793 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:03.821807 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:03.821843 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:03.824901 (+ 3058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.825760 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.825767 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.825769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.825901 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:03.826379 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:03.826384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:03.826386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:03.826773 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:03.827075 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:03.827420 (+ 345us) server_negotiation.cc:299] Negotiation successful
0927 23:07:03.827577 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:07:03.828635 18349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:03.947431 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 18226
I20260927 23:07:03.966663 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 17866
I20260927 23:07:03.967716 17928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:04.083954 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 17866
I20260927 23:07:04.106453 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46819
--webserver_interface=127.3.240.254
--webserver_port=45793
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:04.420603 18403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:04.420836 18403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:04.421029 18403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:04.427831 18403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:07:04.427927 18403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:04.428032 18403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:04.428109 18403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:07:04.428184 18403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:07:04.437304 18403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=45793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:04.438521 18403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:04.439845 18403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:04.461107 18409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:04.461108 18410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:04.461440 18403 server_base.cc:1034] running on GCE node
W20260927 23:07:04.461098 18412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:04.462499 18403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:04.464054 18403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:04.465289 18403 hybrid_clock.cc:648] HybridClock initialized: now 1790550424465255 us; error 53 us; skew 500 ppm
I20260927 23:07:04.465658 18403 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:04 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:04.469055 18403 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:07:04.470568 18403 webserver.cc:460] Webserver started at http://127.3.240.254:45793/ using document root <none> and password file <none>
I20260927 23:07:04.471323 18403 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:04.471436 18403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:04.476995 18403 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:07:04.479727 18419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:04.480995 18403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:07:04.481127 18403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "5952c24970de4118b1df3704a7eb4f91"
format_stamp: "Formatted at 2026-09-27 23:07:01 on dist-test-slave-lmfp"
server_key: "399a2a09d4ca3dd4b1d155c21e9bcf18"
server_key_iv: "7c711b08280303d8907e8cbc8b4175db"
server_key_version: "encryptionkey@0"
I20260927 23:07:04.482102 18403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:04.505975 18403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:04.510528 18403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:04.510748 18403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:04.529199 18403 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46819
I20260927 23:07:04.529222 18470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46819 every 8 connection(s)
I20260927 23:07:04.530637 18403 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:07:04.537374 18471 sys_catalog.cc:263] Verifying existing consensus state
I20260927 23:07:04.537678 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 18403
I20260927 23:07:04.538569 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:43007
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:46819
--webserver_port=34187
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:07:04.542207 18471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Bootstrap starting.
I20260927 23:07:04.561829 18471 log.cc:824] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:04.573271 18471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:07:04.573788 18471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Bootstrap complete.
I20260927 23:07:04.582587 18471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:04.583503 18471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5952c24970de4118b1df3704a7eb4f91, State: Initialized, Role: FOLLOWER
I20260927 23:07:04.584110 18471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:04.584367 18471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:07:04.584476 18471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:07:04.584677 18471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:07:04.588523 18471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:04.588991 18471 leader_election.cc:302] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5952c24970de4118b1df3704a7eb4f91; no voters:
I20260927 23:07:04.589594 18471 leader_election.cc:288] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 23:07:04.589715 18475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:07:04.591919 18471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:07:04.596817 18475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 2 LEADER]: Becoming Leader. State: Replica: 5952c24970de4118b1df3704a7eb4f91, State: Running, Role: LEADER
I20260927 23:07:04.597440 18475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:04.600158 18476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5952c24970de4118b1df3704a7eb4f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } } }
I20260927 23:07:04.600912 18476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:04.606892 18477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5952c24970de4118b1df3704a7eb4f91. Latest consensus state: current_term: 2 leader_uuid: "5952c24970de4118b1df3704a7eb4f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } } }
I20260927 23:07:04.607249 18477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:04.617129 18486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:07:04.626607 18486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:07:04.628378 18486 catalog_manager.cc:1259] Loaded cluster ID: 2f215d23fd5a4f2dab067bcf3e01e41a
I20260927 23:07:04.628551 18486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:07:04.632507 18486 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260927 23:07:04.652807 18486 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260927 23:07:04.661525 18486 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:07:04.663233 18486 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Loaded TSK: 0
I20260927 23:07:04.664599 18486 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 23:07:04.885882 18473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:04.886236 18473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:04.886405 18473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:04.893718 18473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:04.893877 18473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:04.894047 18473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:07:04.904176 18473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:43007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=34187
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:04.905421 18473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:04.906749 18473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:04.921620 18500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:04.921597 18499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:04.921887 18502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:04.923360 18473 server_base.cc:1034] running on GCE node
I20260927 23:07:04.924121 18473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:04.925230 18473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:04.926486 18473 hybrid_clock.cc:648] HybridClock initialized: now 1790550424926429 us; error 43 us; skew 500 ppm
I20260927 23:07:04.926858 18473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:04 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:04.930262 18473 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:07:04.931777 18473 webserver.cc:460] Webserver started at http://127.3.240.193:34187/ using document root <none> and password file <none>
I20260927 23:07:04.932435 18473 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:04.932567 18473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:04.937800 18473 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 23:07:04.940505 18509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:04.941797 18473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 23:07:04.942003 18473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "fe2c285e7e9946e2b7ba2f5053698346"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "eeef7439452a04928bcbfe6c151218d0"
server_key_iv: "4760cb13d542780bc9a663cd92d85cc7"
server_key_version: "encryptionkey@0"
I20260927 23:07:04.942963 18473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:04.984381 18473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:04.988629 18473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:04.988888 18473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:04.989787 18473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:04.991070 18473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:04.991181 18473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:04.991305 18473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:04.991403 18473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:05.022495 18473 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:43007
I20260927 23:07:05.022540 18621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:43007 every 8 connection(s)
I20260927 23:07:05.024006 18473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:07:05.032924 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 18473
I20260927 23:07:05.033553 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:46443
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:46819
--webserver_port=39051
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:07:05 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550424, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:05.047278 18625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.027056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:37945)
0927 23:07:05.027883 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:05.027917 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:05.029564 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:05.030841 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:05.030851 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:05.031449 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:05.031952 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.031968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.033588 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.033598 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.034869 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.034885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.036004 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.036012 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.036290 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.036987 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:05.037008 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:05.039355 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:05.043049 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:05.043058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:05.043073 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:05.043473 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:05.043914 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:05.043919 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:05.043923 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:05.044101 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:05.044816 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:05.044823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:05.045185 (+ 362us) client_negotiation.cc:769] Sending connection context
0927 23:07:05.045427 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 23:07:05.045901 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":342,"threads_started":1}
I20260927 23:07:05.047269 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.026866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:37945 (local address 127.3.240.254:46819)
0927 23:07:05.027500 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:05.027519 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:05.028047 (+ 528us) server_negotiation.cc:407] Connection header received
0927 23:07:05.030079 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:05.030091 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:05.030382 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:05.030716 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:05.032173 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.033428 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.035070 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.035745 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.035973 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.039650 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:05.039663 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:05.039683 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:05.039743 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:05.042882 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:05.043620 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:05.043627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:05.043633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:05.043798 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:05.044220 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:05.044226 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:05.044229 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:05.044664 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:05.044863 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:05.045648 (+ 785us) server_negotiation.cc:299] Negotiation successful
0927 23:07:05.046006 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":271,"threads_started":1}
I20260927 23:07:05.050436 18622 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46819
I20260927 23:07:05.050730 18622 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:05.051445 18622 heartbeater.cc:511] Master 127.3.240.254:46819 requested a full tablet report, sending...
I20260927 23:07:05.053841 18436 ts_manager.cc:194] Registered new tserver with Master: fe2c285e7e9946e2b7ba2f5053698346 (127.3.240.193:43007)
I20260927 23:07:05.056345 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:37945
W20260927 23:07:05.363389 18626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:05.363657 18626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:05.363754 18626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:05.370750 18626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:05.370857 18626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:05.370965 18626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:07:05.380013 18626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:46443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=39051
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:05.381150 18626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:05.382376 18626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:05.396129 18634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:05.396183 18632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:05.396203 18631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:05.397042 18626 server_base.cc:1034] running on GCE node
I20260927 23:07:05.397532 18626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:05.398381 18626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:05.399610 18626 hybrid_clock.cc:648] HybridClock initialized: now 1790550425399560 us; error 42 us; skew 500 ppm
I20260927 23:07:05.399904 18626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:05 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:05.402771 18626 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:07:05.404165 18626 webserver.cc:460] Webserver started at http://127.3.240.194:39051/ using document root <none> and password file <none>
I20260927 23:07:05.404824 18626 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:05.404963 18626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:05.409734 18626 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 23:07:05.412326 18641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:05.413522 18626 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:07:05.413682 18626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "b0b96907e0944127b05425e5559c1444"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "73d5b67bf19d319a5ba54f2c437c0678"
server_key_iv: "bbf0f1702811fb79159cfdbdb64395aa"
server_key_version: "encryptionkey@0"
I20260927 23:07:05.414599 18626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:05.434638 18626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:05.438151 18626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:05.438405 18626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:05.439301 18626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:05.440514 18626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:05.440627 18626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:05.440760 18626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:05.440843 18626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:05.473570 18626 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:46443
I20260927 23:07:05.473629 18753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:46443 every 8 connection(s)
I20260927 23:07:05.474876 18626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:07:05.476635 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 18626
I20260927 23:07:05.477371 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:33987
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:46819
--webserver_port=37647
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:07:05 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:05.498827 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.478429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:49355 (local address 127.3.240.254:46819)
0927 23:07:05.478612 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:05.478619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:05.479083 (+ 464us) server_negotiation.cc:407] Connection header received
0927 23:07:05.482145 (+ 3062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:05.482149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:05.482218 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:05.482317 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:05.484431 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.485265 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.487026 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.487764 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.487911 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.491381 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:05.491389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:05.491397 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:05.491439 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:05.495253 (+ 3814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:05.496042 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:05.496049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:05.496056 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:05.496219 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:05.496670 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:05.496678 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:05.496683 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:05.497143 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:05.497315 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:05.498038 (+ 723us) server_negotiation.cc:299] Negotiation successful
0927 23:07:05.498336 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:07:05.499486 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.478063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:49355)
0927 23:07:05.478892 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:05.478942 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:05.481796 (+ 2854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:05.482601 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:05.482615 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:05.483424 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:05.484225 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.484271 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.485431 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.485435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.486868 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.486881 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.487998 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.488003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.488227 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.488924 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:05.488954 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:05.491159 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:05.495402 (+ 4243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:05.495408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:05.495419 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:05.495911 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:05.496346 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:05.496349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:05.496351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:05.496510 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:05.497277 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:05.497282 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:05.497589 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 23:07:05.497848 (+ 259us) client_negotiation.cc:240] Negotiation successful
0927 23:07:05.498269 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":379,"threads_started":1}
I20260927 23:07:05.501024 18754 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46819
I20260927 23:07:05.501338 18754 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:05.502041 18754 heartbeater.cc:511] Master 127.3.240.254:46819 requested a full tablet report, sending...
I20260927 23:07:05.503423 18436 ts_manager.cc:194] Registered new tserver with Master: b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443)
I20260927 23:07:05.504221 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:49355
W20260927 23:07:05.788713 18757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:05.788973 18757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:05.789125 18757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:05.796285 18757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:05.796411 18757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:05.796588 18757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:07:05.805967 18757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:33987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=37647
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:05.807248 18757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:05.808470 18757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:05.821260 18762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:05.821286 18763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:05.821437 18765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:05.821992 18757 server_base.cc:1034] running on GCE node
I20260927 23:07:05.822655 18757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:05.823573 18757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:05.824793 18757 hybrid_clock.cc:648] HybridClock initialized: now 1790550425824750 us; error 59 us; skew 500 ppm
I20260927 23:07:05.825109 18757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:05 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:05.828043 18757 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:07:05.829473 18757 webserver.cc:460] Webserver started at http://127.3.240.195:37647/ using document root <none> and password file <none>
I20260927 23:07:05.830191 18757 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:05.830353 18757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:05.835668 18757 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:07:05.838101 18772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:05.839382 18757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260927 23:07:05.839540 18757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "2ce1276eca724fafba55c45a82ac0d7a"
format_stamp: "Formatted at 2026-09-27 23:07:03 on dist-test-slave-lmfp"
server_key: "d19413a604a9104e32a4399eeb1b1721"
server_key_iv: "d05f1cf3488f9784079abd84757c4393"
server_key_version: "encryptionkey@0"
I20260927 23:07:05.840457 18757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:05.861958 18757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:05.865554 18757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:05.865805 18757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:05.866660 18757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:05.867913 18757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:05.868041 18757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:05.868166 18757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:05.868247 18757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:05.900348 18757 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33987
I20260927 23:07:05.900426 18884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33987 every 8 connection(s)
I20260927 23:07:05.901659 18757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:07:05.902069 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 18757
Sep 27 23:07:05 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550425, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:05.917742 18624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.903130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.3.240.254:46819)
0927 23:07:05.903405 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:05.903416 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:05.903905 (+ 489us) server_negotiation.cc:407] Connection header received
0927 23:07:05.904125 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:05.904130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:05.904234 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:05.904353 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:05.906055 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.906845 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.908653 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.909069 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.909165 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.912267 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:05.912271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:05.912275 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:05.912301 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:05.915056 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:05.915785 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:05.915790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:05.915794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:05.915946 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:05.916332 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:05.916335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:05.916337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:05.916566 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:05.916730 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:05.917141 (+ 411us) server_negotiation.cc:299] Negotiation successful
0927 23:07:05.917350 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 23:07:05.922761 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.904572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:34197 (local address 127.3.240.254:46819)
0927 23:07:05.905014 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:05.905019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:05.906074 (+ 1055us) server_negotiation.cc:407] Connection header received
0927 23:07:05.908206 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:05.908209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:05.908265 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:05.908362 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:05.910088 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.910837 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.912854 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.913311 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.913467 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.916903 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:05.916907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:05.916910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:05.916932 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:05.919952 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:05.920647 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:05.920652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:05.920655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:05.920773 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:05.921174 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:05.921178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:05.921182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:05.921424 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:05.921577 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:05.922206 (+ 629us) server_negotiation.cc:299] Negotiation successful
0927 23:07:05.922414 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260927 23:07:05.923904 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.905164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:34197)
0927 23:07:05.905921 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:05.905990 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:05.907922 (+ 1932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:05.908633 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:05.908644 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:05.909145 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:05.909903 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.909927 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.911037 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.911044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.912722 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.912733 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.913554 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.913563 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.913802 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.914333 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:05.914352 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:05.916714 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:05.920098 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:05.920105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:05.920125 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:05.920522 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:05.920881 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:05.920886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:05.920891 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:05.921057 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:05.921560 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:05.921567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:05.921864 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 23:07:05.922116 (+ 252us) client_negotiation.cc:240] Negotiation successful
0927 23:07:05.922582 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":279,"threads_started":1}
I20260927 23:07:05.928295 18885 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46819
I20260927 23:07:05.928637 18885 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:05.929600 18885 heartbeater.cc:511] Master 127.3.240.254:46819 requested a full tablet report, sending...
I20260927 23:07:05.931717 18436 ts_manager.cc:194] Registered new tserver with Master: 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987)
I20260927 23:07:05.932651 18436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:34197
I20260927 23:07:05.933786 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:07:05.940217 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.934727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.3.240.254:46819)
0927 23:07:05.934891 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:05.934895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:05.935388 (+ 493us) server_negotiation.cc:407] Connection header received
0927 23:07:05.935516 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:05.935520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:05.935652 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:05.935794 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:07:05.936302 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.936958 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.938297 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.938681 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.938782 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.939276 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:07:05.939762 (+ 486us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:07:05.939866 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:05.939982 (+ 116us) server_negotiation.cc:299] Negotiation successful
0927 23:07:05.940038 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 23:07:05.941042 18436 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:07:05.948416 18436 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:07:05.971508 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.959260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46443 (local address 127.0.0.1:40568)
0927 23:07:05.960134 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:05.960173 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:05.960339 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:05.961456 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:05.961466 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:05.961507 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:05.961999 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.962016 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.964489 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.964497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.967494 (+ 2997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.967513 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.970161 (+ 2648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.970169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.970430 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.970495 (+ 65us) client_negotiation.cc:769] Sending connection context
0927 23:07:05.970657 (+ 162us) client_negotiation.cc:240] Negotiation successful
0927 23:07:05.970905 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":587,"threads_started":1}
I20260927 23:07:05.972863 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.959637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.3.240.194:46443)
0927 23:07:05.960143 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:05.960152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:05.960248 (+ 96us) server_negotiation.cc:407] Connection header received
0927 23:07:05.960527 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:05.960535 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:05.960808 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:05.961050 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:05.962269 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.964290 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.967702 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.969566 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.969712 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.969986 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:05.972172 (+ 2186us) server_negotiation.cc:299] Negotiation successful
0927 23:07:05.972416 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":813,"threads_started":1}
I20260927 23:07:05.977687 18689 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff2304a4e91469abfc106564b417fe5 (DEFAULT_TABLE table=test-table [id=f71c0efb7a0649e5930afc1f8b625c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:05.979193 18689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff2304a4e91469abfc106564b417fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:05.982054 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.959851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33987 (local address 127.0.0.1:47672)
0927 23:07:05.971833 (+ 11982us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:05.971859 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:05.971985 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:05.976866 (+ 4881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:05.976871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:05.976895 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:05.977042 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.977051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.978715 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.978724 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.980074 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.980089 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.981220 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.981244 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.981436 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.981476 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 23:07:05.981587 (+ 111us) client_negotiation.cc:240] Negotiation successful
0927 23:07:05.981712 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11857,"mutex_wait_us":668,"thread_start_us":14923,"threads_started":1}
I20260927 23:07:05.982967 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.960203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.3.240.195:33987)
0927 23:07:05.976052 (+ 15849us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:05.976059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:05.976088 (+ 29us) server_negotiation.cc:407] Connection header received
0927 23:07:05.976167 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:05.976173 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:05.976368 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:05.976560 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:05.977220 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.978587 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.980220 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.981058 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.981200 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.981441 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:05.981625 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 23:07:05.981776 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15697,"thread_start_us":15520,"threads_started":1}
I20260927 23:07:05.983012 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.960580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:55466)
0927 23:07:05.973709 (+ 13129us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:05.973734 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:05.973840 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:05.975548 (+ 1708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:05.975556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:05.975588 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:05.975947 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.975957 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.978010 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.978015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.980129 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:05.980143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.982197 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.982203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:05.982391 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.982422 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 23:07:05.982506 (+ 84us) client_negotiation.cc:240] Negotiation successful
0927 23:07:05.982623 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13021,"mutex_wait_us":53,"thread_start_us":12894,"threads_started":1}
I20260927 23:07:05.983695 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:05.961050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.3.240.193:43007)
0927 23:07:05.974395 (+ 13345us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:05.974408 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:05.974443 (+ 35us) server_negotiation.cc:407] Connection header received
0927 23:07:05.974568 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:05.974580 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:05.974895 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:05.975115 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:05.976165 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.977855 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.980318 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:05.981713 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:05.981849 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:05.982039 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:05.983269 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0927 23:07:05.983422 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13148,"thread_start_us":14262,"threads_started":1}
I20260927 23:07:05.986332 18820 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff2304a4e91469abfc106564b417fe5 (DEFAULT_TABLE table=test-table [id=f71c0efb7a0649e5930afc1f8b625c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:05.987040 18557 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff2304a4e91469abfc106564b417fe5 (DEFAULT_TABLE table=test-table [id=f71c0efb7a0649e5930afc1f8b625c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:05.987720 18820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff2304a4e91469abfc106564b417fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:05.988322 18557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff2304a4e91469abfc106564b417fe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:05.992319 18901 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Bootstrap starting.
I20260927 23:07:05.995182 18901 tablet_bootstrap.cc:654] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:05.996140 18901 log.cc:824] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:05.998909 18901 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: No bootstrap required, opened a new log
I20260927 23:07:05.999188 18901 ts_tablet_manager.cc:1397] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260927 23:07:06.002812 18903 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Bootstrap starting.
I20260927 23:07:06.006199 18903 tablet_bootstrap.cc:654] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:06.006984 18904 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Bootstrap starting.
I20260927 23:07:06.007434 18903 log.cc:824] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:06.007903 18901 raft_consensus.cc:348] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.008339 18901 raft_consensus.cc:374] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:06.008448 18901 raft_consensus.cc:729] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b96907e0944127b05425e5559c1444, State: Initialized, Role: FOLLOWER
I20260927 23:07:06.009064 18901 consensus_queue.cc:260] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.010136 18754 heartbeater.cc:503] Master 127.3.240.254:46819 was elected leader, sending a full tablet report...
I20260927 23:07:06.010890 18901 ts_tablet_manager.cc:1428] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260927 23:07:06.011704 18903 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: No bootstrap required, opened a new log
I20260927 23:07:06.011977 18903 ts_tablet_manager.cc:1397] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260927 23:07:06.012257 18904 tablet_bootstrap.cc:654] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:06.013550 18904 log.cc:824] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:06.016908 18904 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: No bootstrap required, opened a new log
I20260927 23:07:06.017151 18904 ts_tablet_manager.cc:1397] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 23:07:06.023137 18903 raft_consensus.cc:348] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.023468 18903 raft_consensus.cc:374] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:06.023579 18903 raft_consensus.cc:729] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ce1276eca724fafba55c45a82ac0d7a, State: Initialized, Role: FOLLOWER
I20260927 23:07:06.024091 18903 consensus_queue.cc:260] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.025104 18885 heartbeater.cc:503] Master 127.3.240.254:46819 was elected leader, sending a full tablet report...
I20260927 23:07:06.025713 18903 ts_tablet_manager.cc:1428] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260927 23:07:06.028270 18904 raft_consensus.cc:348] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.028721 18904 raft_consensus.cc:374] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:06.028863 18904 raft_consensus.cc:729] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2c285e7e9946e2b7ba2f5053698346, State: Initialized, Role: FOLLOWER
I20260927 23:07:06.029511 18904 consensus_queue.cc:260] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.030637 18622 heartbeater.cc:503] Master 127.3.240.254:46819 was elected leader, sending a full tablet report...
I20260927 23:07:06.031023 18904 ts_tablet_manager.cc:1428] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260927 23:07:06.153350 18886 tablet.cc:2369] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:07:06.221992 18908 raft_consensus.cc:482] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:06.222293 18908 raft_consensus.cc:504] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.223893 18908 leader_election.cc:288] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443), fe2c285e7e9946e2b7ba2f5053698346 (127.3.240.193:43007)
W20260927 23:07:06.226787 18755 tablet.cc:2369] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:07:06.229605 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.224321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46443 (local address 127.3.240.195:52199)
0927 23:07:06.224509 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.224527 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:06.224630 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:06.225024 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:06.225028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:06.225055 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:06.225262 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:06.225275 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.226184 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.226190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:06.227963 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:06.227973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.228942 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.228947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:06.229126 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:06.229147 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 23:07:06.229242 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 23:07:06.229360 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260927 23:07:06.229602 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.224303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:52199 (local address 127.3.240.194:46443)
0927 23:07:06.224532 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:06.224539 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:06.224589 (+ 50us) server_negotiation.cc:407] Connection header received
0927 23:07:06.224774 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:06.224780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:06.224863 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:06.224970 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:06.225404 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.226054 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.228106 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.228800 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.228955 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:06.229023 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:06.229267 (+ 244us) server_negotiation.cc:299] Negotiation successful
0927 23:07:06.229379 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 23:07:06.230846 18910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.224803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.3.240.195:49733)
0927 23:07:06.225573 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.225592 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:06.225745 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:06.226100 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:06.226105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:06.226128 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:06.226565 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:06.226574 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.227739 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.227745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:06.229001 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:06.229011 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.230356 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.230360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:06.230465 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:06.230486 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 23:07:06.230554 (+ 68us) client_negotiation.cc:240] Negotiation successful
0927 23:07:06.230629 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":478,"threads_started":1}
I20260927 23:07:06.230727 18709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff2304a4e91469abfc106564b417fe5" candidate_uuid: "2ce1276eca724fafba55c45a82ac0d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b96907e0944127b05425e5559c1444" is_pre_election: true
I20260927 23:07:06.231087 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.225110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:49733 (local address 127.3.240.193:43007)
0927 23:07:06.225316 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:06.225323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:06.225696 (+ 373us) server_negotiation.cc:407] Connection header received
0927 23:07:06.225887 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:06.225898 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:06.225979 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:06.226085 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:06.226744 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.227560 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.229164 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.230167 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.230334 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:06.230414 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:06.230621 (+ 207us) server_negotiation.cc:299] Negotiation successful
0927 23:07:06.230765 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:07:06.231278 18709 raft_consensus.cc:2454] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce1276eca724fafba55c45a82ac0d7a in term 0.
I20260927 23:07:06.231971 18775 leader_election.cc:302] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce1276eca724fafba55c45a82ac0d7a, b0b96907e0944127b05425e5559c1444; no voters:
I20260927 23:07:06.232122 18577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff2304a4e91469abfc106564b417fe5" candidate_uuid: "2ce1276eca724fafba55c45a82ac0d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2c285e7e9946e2b7ba2f5053698346" is_pre_election: true
I20260927 23:07:06.232389 18908 raft_consensus.cc:2781] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:07:06.232592 18908 raft_consensus.cc:482] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:07:06.232740 18908 raft_consensus.cc:3037] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:06.232676 18577 raft_consensus.cc:2454] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ce1276eca724fafba55c45a82ac0d7a in term 0.
I20260927 23:07:06.235157 18908 raft_consensus.cc:504] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.235913 18908 leader_election.cc:288] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [CANDIDATE]: Term 1 election: Requested vote from peers b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443), fe2c285e7e9946e2b7ba2f5053698346 (127.3.240.193:43007)
I20260927 23:07:06.236497 18709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff2304a4e91469abfc106564b417fe5" candidate_uuid: "2ce1276eca724fafba55c45a82ac0d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0b96907e0944127b05425e5559c1444"
I20260927 23:07:06.236536 18577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ff2304a4e91469abfc106564b417fe5" candidate_uuid: "2ce1276eca724fafba55c45a82ac0d7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe2c285e7e9946e2b7ba2f5053698346"
I20260927 23:07:06.236706 18709 raft_consensus.cc:3037] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:06.236716 18577 raft_consensus.cc:3037] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:06.238909 18577 raft_consensus.cc:2454] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce1276eca724fafba55c45a82ac0d7a in term 1.
I20260927 23:07:06.239059 18709 raft_consensus.cc:2454] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ce1276eca724fafba55c45a82ac0d7a in term 1.
I20260927 23:07:06.239352 18775 leader_election.cc:302] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ce1276eca724fafba55c45a82ac0d7a, fe2c285e7e9946e2b7ba2f5053698346; no voters:
I20260927 23:07:06.239636 18908 raft_consensus.cc:2781] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:07:06.240063 18908 raft_consensus.cc:686] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 LEADER]: Becoming Leader. State: Replica: 2ce1276eca724fafba55c45a82ac0d7a, State: Running, Role: LEADER
I20260927 23:07:06.240507 18908 consensus_queue.cc:237] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:06.245242 18436 catalog_manager.cc:5495] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195). New cstate: current_term: 1 leader_uuid: "2ce1276eca724fafba55c45a82ac0d7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } health_report { overall_health: HEALTHY } } }
I20260927 23:07:06.262032 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.256880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.3.240.195:33987)
0927 23:07:06.257090 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:06.257097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:06.257202 (+ 105us) server_negotiation.cc:407] Connection header received
0927 23:07:06.257448 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:06.257451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:06.257527 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:06.257666 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:07:06.257990 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.258701 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.260066 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.260441 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.260568 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:06.261036 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:07:06.261493 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:07:06.261640 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:06.261696 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 23:07:06.261780 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 23:07:06.269603 18709 raft_consensus.cc:1261] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 1 FOLLOWER]: Refusing update from remote peer 2ce1276eca724fafba55c45a82ac0d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:07:06.269650 18577 raft_consensus.cc:1261] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Refusing update from remote peer 2ce1276eca724fafba55c45a82ac0d7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:07:06.270428 18911 consensus_queue.cc:1035] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:07:06.270741 18908 consensus_queue.cc:1035] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 23:07:06.276237 18623 tablet.cc:2369] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:07:06.289142 18915 mvcc.cc:205] Tried to move back new op lower bound from 7334094545990885376 to 7334094545884729344. Current Snapshot: MvccSnapshot[applied={T|T < 7334094545990885376}]
I20260927 23:07:06.297523 18435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:07:06.297739 18435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:07:06.301349 18435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Sending DeleteTablet for 3 replicas of tablet 9ff2304a4e91469abfc106564b417fe5
I20260927 23:07:06.302452 18820 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff2304a4e91469abfc106564b417fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:06 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:47672
I20260927 23:07:06.302501 18689 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff2304a4e91469abfc106564b417fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:06 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:40568
I20260927 23:07:06.302657 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 18473
I20260927 23:07:06.302697 18557 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ff2304a4e91469abfc106564b417fe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:06 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:55466
I20260927 23:07:06.303645 18924 tablet_replica.cc:330] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: stopping tablet replica
I20260927 23:07:06.304037 18924 raft_consensus.cc:2229] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:07:06.304477 18924 raft_consensus.cc:2258] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:06.306363 18924 ts_tablet_manager.cc:1905] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:06.312784 18924 ts_tablet_manager.cc:1918] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:06.312939 18924 log.cc:1196] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/9ff2304a4e91469abfc106564b417fe5
I20260927 23:07:06.313485 18924 ts_tablet_manager.cc:1939] T 9ff2304a4e91469abfc106564b417fe5 P b0b96907e0944127b05425e5559c1444: Deleting consensus metadata
I20260927 23:07:06.315058 18422 catalog_manager.cc:4841] TS b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443): tablet 9ff2304a4e91469abfc106564b417fe5 (table test-table [id=f71c0efb7a0649e5930afc1f8b625c69]) successfully deleted
I20260927 23:07:06.319244 18925 tablet_replica.cc:330] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: stopping tablet replica
I20260927 23:07:06.319768 18925 raft_consensus.cc:2229] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:07:06.320402 18925 raft_consensus.cc:2258] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:06.320801 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.303528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55472 (local address 127.3.240.193:43007)
0927 23:07:06.303712 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:06.303721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:06.303745 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:07:06.303948 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:06.303956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:06.304066 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:06.304180 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:06.305917 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.307378 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.309336 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.309925 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.310041 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:06.313596 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:06.313611 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:06.313627 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:06.313667 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:06.317302 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:06.318173 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:06.318180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:06.318182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:06.318311 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:06.318758 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:06.318765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:06.318768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:06.319469 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:06.319790 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:06.320091 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 23:07:06.320337 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 23:07:06.321853 18923 tablet_replica.cc:330] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: stopping tablet replica
I20260927 23:07:06.321899 18617 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:06.322319 18923 raft_consensus.cc:2229] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:07:06.322989 18923 raft_consensus.cc:2258] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:06.324980 18923 ts_tablet_manager.cc:1905] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:06.332623 18923 ts_tablet_manager.cc:1918] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:06.332777 18923 log.cc:1196] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/9ff2304a4e91469abfc106564b417fe5
I20260927 23:07:06.333415 18923 ts_tablet_manager.cc:1939] T 9ff2304a4e91469abfc106564b417fe5 P 2ce1276eca724fafba55c45a82ac0d7a: Deleting consensus metadata
I20260927 23:07:06.335397 18423 catalog_manager.cc:4841] TS 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987): tablet 9ff2304a4e91469abfc106564b417fe5 (table test-table [id=f71c0efb7a0649e5930afc1f8b625c69]) successfully deleted
I20260927 23:07:06.413131 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 18473
W20260927 23:07:06.431623 18422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:43007 (error 108)
I20260927 23:07:06.431857 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 18626
I20260927 23:07:06.433434 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.432953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:40388)
0927 23:07:06.433131 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.433265 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260927 23:07:06.433799 18422 catalog_manager.cc:4584] TS fe2c285e7e9946e2b7ba2f5053698346 (127.3.240.193:43007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ff2304a4e91469abfc106564b417fe5: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
I20260927 23:07:06.445815 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.432651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41300 (local address 127.3.240.194:46443)
0927 23:07:06.432878 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:06.432883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:06.433286 (+ 403us) server_negotiation.cc:407] Connection header received
0927 23:07:06.433354 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:06.433359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:06.433436 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:06.433531 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:06.435174 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.435865 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.437117 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.437489 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.437607 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:06.439985 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:06.439998 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:06.440012 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:06.440052 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:06.442827 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:06.443665 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:06.443673 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:06.443676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:06.443800 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:06.444204 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:06.444208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:06.444210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:06.444642 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:06.444957 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:06.445239 (+ 282us) server_negotiation.cc:299] Negotiation successful
0927 23:07:06.445456 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260927 23:07:06.446398 18749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:06.487324 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.486820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:40394)
0927 23:07:06.487034 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.487138 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 23:07:06.488467 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.488047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:40402)
0927 23:07:06.488232 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.488318 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 23:07:06.536147 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 18626
I20260927 23:07:06.549113 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.548450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:40418)
0927 23:07:06.548784 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.548896 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260927 23:07:06.550441 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.549988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:40426)
0927 23:07:06.550193 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.550294 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260927 23:07:06.555356 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 18757
I20260927 23:07:06.574851 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.560403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.3.240.195:33987)
0927 23:07:06.560597 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:06.560606 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:06.560632 (+ 26us) server_negotiation.cc:407] Connection header received
0927 23:07:06.560712 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:06.560716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:06.560803 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:06.560924 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:06.562224 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.563121 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.565133 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:06.565564 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:06.565650 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:06.568948 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:06.568958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:06.568976 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:06.569015 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:06.571898 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:06.572797 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:06.572804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:06.572806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:06.572961 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:06.573379 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:06.573385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:06.573387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:06.573758 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:06.573995 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:06.574326 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 23:07:06.574478 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 23:07:06.575630 18880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:06.631067 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.630375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:40442)
0927 23:07:06.630673 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.630792 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260927 23:07:06.632213 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.631760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:40458)
0927 23:07:06.631971 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.632062 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260927 23:07:06.696395 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 18757
I20260927 23:07:06.716212 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 18403
I20260927 23:07:06.717483 18466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:06.829775 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:06.829140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43007 (local address 127.0.0.1:40474)
0927 23:07:06.829475 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:06.829582 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:43007: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260927 23:07:06.829808 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 18403
I20260927 23:07:06.851257 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46819
--webserver_interface=127.3.240.254
--webserver_port=45793
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46819
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:07.177505 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:07.177817 18931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:07.177978 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:07.184748 18931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:07:07.184878 18931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:07.184985 18931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:07.185088 18931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:07:07.185178 18931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:07:07.194487 18931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=45793
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:07.195889 18931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:07.197202 18931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:07.208493 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:07.208673 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:07.208840 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:07.209538 18931 server_base.cc:1034] running on GCE node
I20260927 23:07:07.210278 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:07.211583 18931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:07.212767 18931 hybrid_clock.cc:648] HybridClock initialized: now 1790550427212717 us; error 52 us; skew 500 ppm
I20260927 23:07:07.213073 18931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:07 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:07.223199 18931 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:07:07.224774 18931 webserver.cc:460] Webserver started at http://127.3.240.254:45793/ using document root <none> and password file <none>
I20260927 23:07:07.225463 18931 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:07.225602 18931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:07.230918 18931 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:07:07.233587 18946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:07.234813 18931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260927 23:07:07.235060 18931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "5952c24970de4118b1df3704a7eb4f91"
format_stamp: "Formatted at 2026-09-27 23:07:01 on dist-test-slave-lmfp"
server_key: "399a2a09d4ca3dd4b1d155c21e9bcf18"
server_key_iv: "7c711b08280303d8907e8cbc8b4175db"
server_key_version: "encryptionkey@0"
I20260927 23:07:07.236061 18931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:07.268150 18931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:07.272810 18931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:07.273118 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:07.292773 18931 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46819
I20260927 23:07:07.292778 18997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46819 every 8 connection(s)
I20260927 23:07:07.294337 18931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:07:07.301049 18998 sys_catalog.cc:263] Verifying existing consensus state
I20260927 23:07:07.302532 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 18931
I20260927 23:07:07.303449 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:43007
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:46819
--webserver_port=34187
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:07:07.304292 18998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Bootstrap starting.
I20260927 23:07:07.329946 18998 log.cc:824] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:07.350591 18998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:07:07.351234 18998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91: Bootstrap complete.
I20260927 23:07:07.364959 18998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:07.366339 18998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5952c24970de4118b1df3704a7eb4f91, State: Initialized, Role: FOLLOWER
I20260927 23:07:07.367159 18998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:07.367487 18998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:07:07.367625 18998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:07:07.367834 18998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:07:07.373767 18998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:07.374341 18998 leader_election.cc:302] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5952c24970de4118b1df3704a7eb4f91; no voters:
I20260927 23:07:07.375005 18998 leader_election.cc:288] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 23:07:07.375172 19002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:07:07.376461 19002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [term 3 LEADER]: Becoming Leader. State: Replica: 5952c24970de4118b1df3704a7eb4f91, State: Running, Role: LEADER
I20260927 23:07:07.377060 18998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:07:07.376974 19002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } }
I20260927 23:07:07.379400 19004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5952c24970de4118b1df3704a7eb4f91. Latest consensus state: current_term: 3 leader_uuid: "5952c24970de4118b1df3704a7eb4f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } } }
I20260927 23:07:07.379765 19004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:07.380321 19003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5952c24970de4118b1df3704a7eb4f91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5952c24970de4118b1df3704a7eb4f91" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46819 } } }
I20260927 23:07:07.380570 19003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5952c24970de4118b1df3704a7eb4f91 [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:07.390767 19010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:07:07.400743 19010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:07:07.402588 19010 catalog_manager.cc:1259] Loaded cluster ID: 2f215d23fd5a4f2dab067bcf3e01e41a
I20260927 23:07:07.402724 19010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260927 23:07:07.405179 19010 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
I20260927 23:07:07.407473 19019 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f17d43fa9ad google::LogMessage::Fail() at ??:0
@ 0x7f17d43fd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f17d43fa4de google::LogMessage::Flush() at ??:0
@ 0x7f17d43fba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f17e9e2d512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f17e9e2a976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f17e9eb2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f17e9eb2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f17d7046345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f17d704991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f17d7049799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f17d7024eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f17da2fe6db start_thread at ??:0
@ 0x7f17d329461f clone at ??:0
W20260927 23:07:07.726606 19000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:07.726903 19000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:07.727103 19000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:07.734004 19000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:07.734141 19000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:07.734323 19000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:07:07.743732 19000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:43007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=34187
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:07.745118 19000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:07.746465 19000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:07.759793 19042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:07.759845 19040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:07.759883 19039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:07.760718 19000 server_base.cc:1034] running on GCE node
I20260927 23:07:07.761317 19000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:07.762440 19000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:07.763669 19000 hybrid_clock.cc:648] HybridClock initialized: now 1790550427763616 us; error 44 us; skew 500 ppm
I20260927 23:07:07.763954 19000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:07 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:07.766899 19000 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:07:07.768290 19000 webserver.cc:460] Webserver started at http://127.3.240.193:34187/ using document root <none> and password file <none>
I20260927 23:07:07.768872 19000 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:07.768966 19000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:07.773841 19000 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260927 23:07:07.778317 19049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:07.779666 19000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:07:07.779841 19000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "fe2c285e7e9946e2b7ba2f5053698346"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "eeef7439452a04928bcbfe6c151218d0"
server_key_iv: "4760cb13d542780bc9a663cd92d85cc7"
server_key_version: "encryptionkey@0"
I20260927 23:07:07.780771 19000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:07.807925 19000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:07.812026 19000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:07.812309 19000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:07.813373 19000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:07.815156 19056 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260927 23:07:07.820367 19000 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260927 23:07:07.820461 19000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260927 23:07:07.820621 19000 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260927 23:07:07.823727 19000 ts_tablet_manager.cc:610] Registered 1 tablets
I20260927 23:07:07.823801 19000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260927 23:07:07.824167 19056 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Bootstrap starting.
I20260927 23:07:07.832063 19056 log.cc:824] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:07.846278 19056 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:07:07.846953 19056 tablet_bootstrap.cc:492] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Bootstrap complete.
I20260927 23:07:07.847501 19056 ts_tablet_manager.cc:1397] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.007s
I20260927 23:07:07.855680 19056 raft_consensus.cc:348] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:07.856930 19056 raft_consensus.cc:729] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe2c285e7e9946e2b7ba2f5053698346, State: Initialized, Role: FOLLOWER
I20260927 23:07:07.857690 19056 consensus_queue.cc:260] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:07.859848 19056 ts_tablet_manager.cc:1428] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260927 23:07:07.865347 19000 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:43007
I20260927 23:07:07.865381 19164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:43007 every 8 connection(s)
I20260927 23:07:07.866467 19000 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:07:07.869577 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 19000
I20260927 23:07:07.870330 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:46443
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:46819
--webserver_port=39051
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:07:07.870908 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:07.868906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:48953)
0927 23:07:07.869775 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:07.870020 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":379,"threads_started":1}
W20260927 23:07:07.871507 19165 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:46819 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:46819: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
I20260927 23:07:07.872293 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:07.871977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:41215)
0927 23:07:07.872110 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:07.872176 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 23:07:07.873102 19167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:07.872788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:35381)
0927 23:07:07.872922 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:07.872998 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260927 23:07:07.873368 19165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 23:07:08.184499 19168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:08.184772 19168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:08.184865 19168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:08.191939 19168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:08.192052 19168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:08.192185 19168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:07:08.202090 19168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:46443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=39051
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:08.203313 19168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:08.204501 19168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:08.217476 19176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:08.217564 19173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:08.217507 19174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:08.218156 19168 server_base.cc:1034] running on GCE node
I20260927 23:07:08.218744 19168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:08.219686 19168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:08.220907 19168 hybrid_clock.cc:648] HybridClock initialized: now 1790550428220864 us; error 56 us; skew 500 ppm
I20260927 23:07:08.221204 19168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:08 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550428, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:08.224195 19168 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:07:08.225518 19168 webserver.cc:460] Webserver started at http://127.3.240.194:39051/ using document root <none> and password file <none>
I20260927 23:07:08.226145 19168 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:08.226280 19168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:08.231103 19168 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:07:08.233642 19183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:08.234840 19168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:07:08.235076 19168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "b0b96907e0944127b05425e5559c1444"
format_stamp: "Formatted at 2026-09-27 23:07:02 on dist-test-slave-lmfp"
server_key: "73d5b67bf19d319a5ba54f2c437c0678"
server_key_iv: "bbf0f1702811fb79159cfdbdb64395aa"
server_key_version: "encryptionkey@0"
I20260927 23:07:08.236001 19168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:08.258658 19168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:08.262594 19168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:08.262866 19168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:08.263824 19168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:08.265116 19168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:08.265235 19168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:08.265363 19168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:08.265450 19168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:08.297763 19168 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:46443
I20260927 23:07:08.297843 19295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:46443 every 8 connection(s)
I20260927 23:07:08.299211 19168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:07:08.301151 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 19168
I20260927 23:07:08.301801 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:33987
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:46819
--webserver_port=37647
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:07:08.304229 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:08.302207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:47789)
0927 23:07:08.303031 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:08.303330 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":324,"threads_started":1}
W20260927 23:07:08.304888 19296 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:46819 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:46819: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
I20260927 23:07:08.305734 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:08.305417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:52885)
0927 23:07:08.305568 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:08.305632 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 23:07:08.306519 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:08.306210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:37935)
0927 23:07:08.306337 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:08.306418 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 23:07:08.306712 19296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 23:07:08.607836 19299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:08.608112 19299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:08.608310 19299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:08.615371 19299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:08.615458 19299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:08.615633 19299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:07:08.624533 19299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:33987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=37647
--tserver_master_addrs=127.3.240.254:46819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:08.625694 19299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:08.626914 19299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:08.639174 19304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:08.639184 19305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:08.639279 19307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:08.640849 19299 server_base.cc:1034] running on GCE node
I20260927 23:07:08.641459 19299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:08.642362 19299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:08.643579 19299 hybrid_clock.cc:648] HybridClock initialized: now 1790550428643540 us; error 55 us; skew 500 ppm
I20260927 23:07:08.643879 19299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:08 dist-test-slave-lmfp krb5kdc[17853](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550428, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:08.646929 19299 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:07:08.648274 19299 webserver.cc:460] Webserver started at http://127.3.240.195:37647/ using document root <none> and password file <none>
I20260927 23:07:08.648949 19299 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:08.649086 19299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:08.654014 19299 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260927 23:07:08.656620 19314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:08.657872 19299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 23:07:08.658047 19299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "2ce1276eca724fafba55c45a82ac0d7a"
format_stamp: "Formatted at 2026-09-27 23:07:03 on dist-test-slave-lmfp"
server_key: "d19413a604a9104e32a4399eeb1b1721"
server_key_iv: "d05f1cf3488f9784079abd84757c4393"
server_key_version: "encryptionkey@0"
I20260927 23:07:08.658970 19299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:08.670229 19299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:08.674324 19299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:08.674587 19299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:08.675524 19299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:08.676739 19299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:08.676864 19299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:08.676991 19299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:08.677075 19299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:08.709370 19299 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33987
I20260927 23:07:08.709455 19426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33987 every 8 connection(s)
I20260927 23:07:08.710759 19299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:07:08.712734 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 19299
I20260927 23:07:08.715895 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:08.713870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:53597)
0927 23:07:08.714753 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:08.715036 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":364,"threads_started":1}
W20260927 23:07:08.716475 19427 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:46819 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:46819: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
I20260927 23:07:08.717243 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:08.716919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:56823)
0927 23:07:08.717054 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:08.717127 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 23:07:08.718047 19430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:08.717748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:56005)
0927 23:07:08.717879 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:08.717953 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 23:07:08.718277 19427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 23:07:08.874590 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:08.873916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:53399)
0927 23:07:08.874333 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:08.874411 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":193,"threads_started":1}
I20260927 23:07:09.192941 19432 raft_consensus.cc:482] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:09.193267 19432 raft_consensus.cc:504] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:09.194809 19432 leader_election.cc:288] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443), 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987)
Sep 27 23:07:09 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
Sep 27 23:07:09 dist-test-slave-lmfp krb5kdc[17853](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550427, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:07:09.218015 19434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:09.195412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:43903 (local address 127.3.240.194:46443)
0927 23:07:09.197074 (+ 1662us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:09.197093 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:09.197136 (+ 43us) server_negotiation.cc:407] Connection header received
0927 23:07:09.199174 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:09.199187 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:09.199654 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:09.200003 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:09.202444 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:09.204634 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:09.205966 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:09.206897 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:09.207121 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:09.211507 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:09.211514 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:09.211526 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:09.211557 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:09.214327 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:09.215244 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:09.215253 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:09.215262 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:09.215449 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:09.215891 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:09.215895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:09.215899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:09.216369 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:09.216553 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:09.217284 (+ 731us) server_negotiation.cc:299] Negotiation successful
0927 23:07:09.217639 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1490,"thread_start_us":1379,"threads_started":1}
I20260927 23:07:09.218020 19433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:09.195229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:46443 (local address 127.3.240.193:43903)
0927 23:07:09.196817 (+ 1588us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:09.196841 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:09.196943 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:09.200175 (+ 3232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:09.200187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:09.201104 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:09.202238 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:09.202264 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:09.204791 (+ 2527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:09.204794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:09.205832 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:09.205845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:09.207178 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:09.207186 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:09.207510 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:09.208414 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:09.208451 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:09.211239 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:09.214464 (+ 3225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:09.214472 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:09.214485 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:09.215076 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:09.215574 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:09.215579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:09.215583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:09.215770 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:09.216514 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:09.216521 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:09.216816 (+ 295us) client_negotiation.cc:769] Sending connection context
0927 23:07:09.217059 (+ 243us) client_negotiation.cc:240] Negotiation successful
0927 23:07:09.217549 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1452,"mutex_wait_us":368,"spinlock_wait_cycles":52480,"thread_start_us":1156,"threads_started":1}
I20260927 23:07:09.218032 19435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:09.195460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:45177 (local address 127.3.240.195:33987)
0927 23:07:09.197393 (+ 1933us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:09.197408 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:09.197447 (+ 39us) server_negotiation.cc:407] Connection header received
0927 23:07:09.199398 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:09.199408 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:09.199729 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:09.200003 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:09.202449 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:09.204292 (+ 1843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:09.206012 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:09.206897 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:09.207121 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:09.211628 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:09.211634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:09.211645 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:09.211679 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:09.214604 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:09.215462 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:09.215469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:09.215476 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:09.215635 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:09.216035 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:09.216040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:09.216043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:09.216481 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:09.216660 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:09.217278 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 23:07:09.217636 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1774,"thread_start_us":1654,"threads_started":1}
I20260927 23:07:09.218094 19431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:09.195201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33987 (local address 127.3.240.193:45177)
0927 23:07:09.195349 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:09.195383 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:09.196063 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:09.200175 (+ 4112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:09.200187 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:09.201104 (+ 917us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:09.202238 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:09.202264 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:09.204467 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:09.204472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:09.205830 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:09.205842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:09.207220 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:09.207227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:09.207512 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:09.208414 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:09.208444 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:09.211408 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:09.214742 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:09.214750 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:09.214755 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:09.215286 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:09.215757 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:09.215762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:09.215766 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:09.215923 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:09.216602 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:09.216607 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:09.216818 (+ 211us) client_negotiation.cc:769] Sending connection context
0927 23:07:09.217060 (+ 242us) client_negotiation.cc:240] Negotiation successful
0927 23:07:09.217698 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":401,"spinlock_wait_cycles":27520}
W20260927 23:07:09.221041 19053 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
W20260927 23:07:09.221321 19052 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
I20260927 23:07:09.221474 19052 leader_election.cc:302] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe2c285e7e9946e2b7ba2f5053698346; no voters: 2ce1276eca724fafba55c45a82ac0d7a, b0b96907e0944127b05425e5559c1444
I20260927 23:07:09.221851 19432 raft_consensus.cc:2726] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:07:09.307932 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:09.307196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:37013)
0927 23:07:09.307580 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:09.307717 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":176,"threads_started":1}
I20260927 23:07:09.719549 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:09.718716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:35263)
0927 23:07:09.719227 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:09.719378 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":209,"threads_started":1}
I20260927 23:07:09.876134 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:09.875553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:56309)
0927 23:07:09.875926 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:09.875996 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":188,"threads_started":1}
I20260927 23:07:10.309381 19439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:10.308734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:50599)
0927 23:07:10.309145 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:10.309228 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":207,"threads_started":1}
I20260927 23:07:10.720984 19440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:10.720343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:33695)
0927 23:07:10.720730 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:10.720800 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":214,"threads_started":1}
I20260927 23:07:10.869484 19441 raft_consensus.cc:482] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:10.869688 19441 raft_consensus.cc:504] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:10.870293 19441 leader_election.cc:288] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443), 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987)
W20260927 23:07:10.871244 19053 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
W20260927 23:07:10.871438 19052 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
I20260927 23:07:10.871552 19052 leader_election.cc:302] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe2c285e7e9946e2b7ba2f5053698346; no voters: 2ce1276eca724fafba55c45a82ac0d7a, b0b96907e0944127b05425e5559c1444
I20260927 23:07:10.871747 19441 raft_consensus.cc:2726] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:07:10.877628 19442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:10.876877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:43411)
0927 23:07:10.877383 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:10.877466 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":262,"threads_started":1}
I20260927 23:07:11.310674 19443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:11.310110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:44195)
0927 23:07:11.310469 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:11.310539 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":185,"threads_started":1}
I20260927 23:07:11.722411 19444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:11.721768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:53993)
0927 23:07:11.722193 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:11.722262 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":254,"threads_started":1}
I20260927 23:07:11.879052 19445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:11.878386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:57447)
0927 23:07:11.878784 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:11.878859 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":225,"threads_started":1}
I20260927 23:07:12.312439 19446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:12.311542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:48903)
0927 23:07:12.312086 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:12.312193 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":295,"threads_started":1}
I20260927 23:07:12.724078 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:12.723346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:59789)
0927 23:07:12.723810 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:12.723890 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":231,"threads_started":1}
I20260927 23:07:12.880709 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:12.879950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:44523)
0927 23:07:12.880426 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:12.880528 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":233,"threads_started":1}
I20260927 23:07:13.313911 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:13.313280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:46145)
0927 23:07:13.313667 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:13.313750 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":209,"threads_started":1}
I20260927 23:07:13.486094 19450 raft_consensus.cc:482] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:13.486280 19450 raft_consensus.cc:504] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:13.486943 19450 leader_election.cc:288] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443), 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987)
W20260927 23:07:13.487898 19052 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
W20260927 23:07:13.488077 19053 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
I20260927 23:07:13.488178 19053 leader_election.cc:302] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe2c285e7e9946e2b7ba2f5053698346; no voters: 2ce1276eca724fafba55c45a82ac0d7a, b0b96907e0944127b05425e5559c1444
I20260927 23:07:13.488431 19450 raft_consensus.cc:2726] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:07:13.725654 19451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:13.724960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:40757)
0927 23:07:13.725413 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:13.725488 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260927 23:07:13.882342 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:13.881542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:58213)
0927 23:07:13.882010 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:13.882107 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":227,"threads_started":1}
I20260927 23:07:14.315466 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:14.314755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:54659)
0927 23:07:14.315228 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:14.315315 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":247,"threads_started":1}
I20260927 23:07:14.727308 19454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:14.726614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:42975)
0927 23:07:14.727053 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:14.727125 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
I20260927 23:07:14.884001 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:14.883226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:39205)
0927 23:07:14.883624 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:14.883724 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":200,"threads_started":1}
I20260927 23:07:15.317185 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:15.316398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:33033)
0927 23:07:15.316914 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:15.317001 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":248,"threads_started":1}
I20260927 23:07:15.728930 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:15.728104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:45745)
0927 23:07:15.728667 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:15.728738 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":301,"threads_started":1}
I20260927 23:07:15.885496 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:15.884769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:46935)
0927 23:07:15.885264 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:15.885348 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":309,"threads_started":1}
I20260927 23:07:16.319039 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:16.318206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:52329)
0927 23:07:16.318722 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:16.318799 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":303,"threads_started":1}
I20260927 23:07:16.730649 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:16.729967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:57015)
0927 23:07:16.730398 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:16.730478 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
I20260927 23:07:16.886924 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:16.886260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:44693)
0927 23:07:16.886664 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:16.886751 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":211,"threads_started":1}
I20260927 23:07:17.320760 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:17.319871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:58379)
0927 23:07:17.320384 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:17.320521 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":252,"threads_started":1}
I20260927 23:07:17.714023 19463 raft_consensus.cc:482] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:17.714186 19463 raft_consensus.cc:504] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:17.714775 19463 leader_election.cc:288] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443), 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987)
W20260927 23:07:17.715689 19053 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
W20260927 23:07:17.715864 19052 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
I20260927 23:07:17.715993 19052 leader_election.cc:302] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe2c285e7e9946e2b7ba2f5053698346; no voters: 2ce1276eca724fafba55c45a82ac0d7a, b0b96907e0944127b05425e5559c1444
I20260927 23:07:17.716260 19463 raft_consensus.cc:2726] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:07:17.732121 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:17.731434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:35099)
0927 23:07:17.731906 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:17.731982 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":249,"threads_started":1}
I20260927 23:07:17.888612 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:17.887817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:38325)
0927 23:07:17.888308 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:17.888423 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":243,"threads_started":1}
I20260927 23:07:18.322562 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:18.321727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:49317)
0927 23:07:18.322191 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:18.322359 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":269,"threads_started":1}
I20260927 23:07:18.733579 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:18.732937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:50853)
0927 23:07:18.733346 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:18.733418 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":234,"threads_started":1}
I20260927 23:07:18.890192 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:18.889415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:56879)
0927 23:07:18.889905 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:18.890004 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":235,"threads_started":1}
I20260927 23:07:19.324127 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:19.323423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:33487)
0927 23:07:19.323898 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:19.323979 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":249,"threads_started":1}
I20260927 23:07:19.735302 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:19.734374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:55507)
0927 23:07:19.735060 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:19.735158 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":393,"threads_started":1}
I20260927 23:07:19.891808 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:19.891114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:33465)
0927 23:07:19.891552 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:19.891643 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260927 23:07:20.325915 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:20.325127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:47747)
0927 23:07:20.325650 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:20.325733 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":293,"threads_started":1}
I20260927 23:07:20.736850 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:20.736157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:59623)
0927 23:07:20.736604 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:20.736683 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":230,"threads_started":1}
I20260927 23:07:20.893180 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:20.892537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:49919)
0927 23:07:20.892938 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:20.893013 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":226,"threads_started":1}
I20260927 23:07:21.327703 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:21.326830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:34841)
0927 23:07:21.327416 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:21.327541 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":311,"threads_started":1}
I20260927 23:07:21.710208 19476 raft_consensus.cc:482] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:21.710441 19476 raft_consensus.cc:504] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0b96907e0944127b05425e5559c1444" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 46443 } } peers { permanent_uuid: "fe2c285e7e9946e2b7ba2f5053698346" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43007 } } peers { permanent_uuid: "2ce1276eca724fafba55c45a82ac0d7a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33987 } }
I20260927 23:07:21.711279 19476 leader_election.cc:288] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443), 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987)
W20260927 23:07:21.712182 19053 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2ce1276eca724fafba55c45a82ac0d7a (127.3.240.195:33987): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
W20260927 23:07:21.712350 19052 leader_election.cc:341] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b0b96907e0944127b05425e5559c1444 (127.3.240.194:46443): Not found: Tablet not found: 9ff2304a4e91469abfc106564b417fe5
I20260927 23:07:21.712530 19052 leader_election.cc:302] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe2c285e7e9946e2b7ba2f5053698346; no voters: 2ce1276eca724fafba55c45a82ac0d7a, b0b96907e0944127b05425e5559c1444
I20260927 23:07:21.712761 19476 raft_consensus.cc:2726] T 9ff2304a4e91469abfc106564b417fe5 P fe2c285e7e9946e2b7ba2f5053698346 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260927 23:07:21.738313 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:21.737651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:58531)
0927 23:07:21.738076 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:21.738171 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":229,"threads_started":1}
I20260927 23:07:21.894671 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:21.893961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:52389)
0927 23:07:21.894356 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:21.894452 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260927 23:07:22.329255 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:22.328554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:43363)
0927 23:07:22.328988 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:22.329066 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":213,"threads_started":1}
I20260927 23:07:22.740100 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:22.739243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:49099)
0927 23:07:22.739676 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:22.739791 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":228,"threads_started":1}
I20260927 23:07:22.896342 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:22.895692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.193:60347)
0927 23:07:22.896077 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:22.896189 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":211,"threads_started":1}
I20260927 23:07:23.331100 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:23.330298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.194:55513)
0927 23:07:23.330728 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:23.330842 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":209,"threads_started":1}
I20260927 23:07:23.720129 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 19000
I20260927 23:07:23.734897 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:23.721156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51608 (local address 127.3.240.193:43007)
0927 23:07:23.721783 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:23.721793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:23.721826 (+ 33us) server_negotiation.cc:407] Connection header received
0927 23:07:23.721906 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:23.721916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:23.722250 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:23.722445 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:23.724155 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:23.725090 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:23.726252 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:23.726800 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:23.726892 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:23.729453 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:23.729464 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:23.729477 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:23.729510 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:23.732110 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:23.732774 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:23.732782 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:23.732785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:23.732963 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:23.733351 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:23.733356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:23.733359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:23.733729 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:23.734024 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:23.734333 (+ 309us) server_negotiation.cc:299] Negotiation successful
0927 23:07:23.734596 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":234,"threads_started":1}
I20260927 23:07:23.736191 19160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:23.742231 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:23.741195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46819 (local address 127.3.240.195:37075)
0927 23:07:23.741812 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:23.741976 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":271,"threads_started":1}
I20260927 23:07:23.825842 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 19000
I20260927 23:07:23.843417 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 19168
I20260927 23:07:23.856889 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:23.844141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.3.240.194:46443)
0927 23:07:23.844651 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:23.844655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:23.844672 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:07:23.844727 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:23.844731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:23.844790 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:23.844883 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:23.846346 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:23.847233 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:23.848445 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:23.848881 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:23.848984 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:23.851799 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:23.851804 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:23.851807 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:23.851835 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:23.854483 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:23.855177 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:23.855181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:23.855183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:23.855291 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:23.855665 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:23.855668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:23.855669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:23.855896 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:23.856088 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:23.856314 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 23:07:23.856470 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":284,"threads_started":1}
I20260927 23:07:23.857566 19291 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:23.941211 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 19168
I20260927 23:07:23.959295 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 19299
I20260927 23:07:23.973125 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:23.960127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.3.240.195:33987)
0927 23:07:23.960705 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:23.960710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:23.960725 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:07:23.960777 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:23.960781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:23.960848 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:23.960974 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:23.962469 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:23.963410 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:23.964796 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:23.965227 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:23.965364 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:23.968095 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:23.968100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:23.968104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:23.968134 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:23.970490 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:23.971213 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:23.971218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:23.971222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:23.971367 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:23.971739 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:23.971744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:23.971747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:23.971986 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:23.972269 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:23.972539 (+ 270us) server_negotiation.cc:299] Negotiation successful
0927 23:07:23.972758 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":275,"threads_started":1}
I20260927 23:07:23.973783 19422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:24.056944 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 19299
2026-09-27T23:07:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24129 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:07:24 dist-test-slave-lmfp krb5kdc[19492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:07:24 dist-test-slave-lmfp krb5kdc[19492](info): set up 2 sockets
Sep 27 23:07:24 dist-test-slave-lmfp krb5kdc[19492](info): commencing operation
krb5kdc: starting...
W20260927 23:07:25.123565 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:07:25 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:07:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:07:25.344899 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45111
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:25.652407 19505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:25.652654 19505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:25.652829 19505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:25.659484 19505 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 23:07:25.659564 19505 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 23:07:25.659662 19505 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 23:07:25.659749 19505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:07:25.659839 19505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:25.659953 19505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:25.660027 19505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:07:25.660118 19505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:07:25.669024 19505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:25.670176 19505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:25.671373 19505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:25.682596 19510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:25.682655 19511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:25.682655 19513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:25.683465 19505 server_base.cc:1034] running on GCE node
I20260927 23:07:25.684146 19505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:25.685375 19505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:25.686625 19505 hybrid_clock.cc:648] HybridClock initialized: now 1790550445686543 us; error 48 us; skew 500 ppm
I20260927 23:07:25.686962 19505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:25 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550445, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:25.690140 19505 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:07:25.691459 19505 webserver.cc:460] Webserver started at http://127.3.240.254:35639/ using document root <none> and password file <none>
I20260927 23:07:25.692061 19505 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:25.692142 19505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:25.692378 19505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:07:25.695200 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "14c7b77c729849479fcbc2041e851bbf"
format_stamp: "Formatted at 2026-09-27 23:07:25 on dist-test-slave-lmfp"
server_key: "ff853b49d2514f15f24971097c29ee6c"
server_key_iv: "920ddc82cd9d8b458cc8764f3cecf915"
server_key_version: "encryptionkey@0"
I20260927 23:07:25.695897 19505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "14c7b77c729849479fcbc2041e851bbf"
format_stamp: "Formatted at 2026-09-27 23:07:25 on dist-test-slave-lmfp"
server_key: "ff853b49d2514f15f24971097c29ee6c"
server_key_iv: "920ddc82cd9d8b458cc8764f3cecf915"
server_key_version: "encryptionkey@0"
I20260927 23:07:25.700577 19505 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:07:25.703720 19520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:25.704916 19505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:07:25.705087 19505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "14c7b77c729849479fcbc2041e851bbf"
format_stamp: "Formatted at 2026-09-27 23:07:25 on dist-test-slave-lmfp"
server_key: "ff853b49d2514f15f24971097c29ee6c"
server_key_iv: "920ddc82cd9d8b458cc8764f3cecf915"
server_key_version: "encryptionkey@0"
I20260927 23:07:25.705286 19505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:25.730903 19505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:25.735164 19505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:25.735396 19505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:25.752764 19505 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:45111
I20260927 23:07:25.752784 19571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:45111 every 8 connection(s)
I20260927 23:07:25.754094 19505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:07:25.754530 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 19505
I20260927 23:07:25.754753 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:07:25.755604 4035 external_mini_cluster.cc:1428] Setting key d5af1163f87b653fd8635b235603c446
I20260927 23:07:25.758222 19572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 23:07:25 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:25.770869 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Bootstrap starting.
I20260927 23:07:25.774891 19572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:25.776156 19572 log.cc:824] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:25.779333 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:25.757462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.3.240.254:45111)
0927 23:07:25.758233 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:25.758244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:25.758282 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:07:25.759923 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:25.759943 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:25.760299 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:25.760628 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:25.762377 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:25.763778 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:25.765663 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:25.766278 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:25.766408 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:25.770970 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:25.770982 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:25.771030 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:25.771112 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:25.774805 (+ 3693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:25.775568 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:25.775583 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:25.775591 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:25.775799 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:25.776165 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:25.776173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:25.776178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:25.776879 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:25.777107 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:25.777647 (+ 540us) server_negotiation.cc:299] Negotiation successful
0927 23:07:25.778035 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":363,"threads_started":1}
I20260927 23:07:25.779482 19572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: No bootstrap required, opened a new log
I20260927 23:07:25.787335 19572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:07:25.787631 19572 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:25.787703 19572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c7b77c729849479fcbc2041e851bbf, State: Initialized, Role: FOLLOWER
I20260927 23:07:25.788127 19572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:07:25.788303 19572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:07:25.788393 19572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:07:25.788573 19572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:25.790935 19572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:07:25.791409 19572 leader_election.cc:302] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14c7b77c729849479fcbc2041e851bbf; no voters:
I20260927 23:07:25.791868 19572 leader_election.cc:288] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:07:25.792025 19577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:07:25.792667 19577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 1 LEADER]: Becoming Leader. State: Replica: 14c7b77c729849479fcbc2041e851bbf, State: Running, Role: LEADER
I20260927 23:07:25.793197 19577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:07:25.793826 19572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:07:25.795727 19578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14c7b77c729849479fcbc2041e851bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } } }
I20260927 23:07:25.796099 19578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:25.797823 19583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:07:25.797585 19579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14c7b77c729849479fcbc2041e851bbf. Latest consensus state: current_term: 1 leader_uuid: "14c7b77c729849479fcbc2041e851bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } } }
I20260927 23:07:25.797986 19579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:25.802861 19583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:07:25.811686 19583 catalog_manager.cc:1347] Generated new cluster ID: f9df7fb6922640eebaff7a77012d082c
I20260927 23:07:25.811801 19583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:07:25.837405 19583 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:07:25.838747 19583 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:07:25.850544 19583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Generated new TSK 0
I20260927 23:07:25.851460 19583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:07:25.914052 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45111
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:26.238045 19598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:26.238355 19598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:26.238485 19598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:26.245401 19598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:26.245487 19598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:26.245591 19598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:07:26.254688 19598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:26.255824 19598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:26.257001 19598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:26.269439 19606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:26.269536 19604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:26.269785 19603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:26.270572 19598 server_base.cc:1034] running on GCE node
I20260927 23:07:26.271165 19598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:26.272019 19598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:26.273238 19598 hybrid_clock.cc:648] HybridClock initialized: now 1790550446273193 us; error 37 us; skew 500 ppm
I20260927 23:07:26.273520 19598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:26 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:26.281957 19598 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:07:26.283473 19598 webserver.cc:460] Webserver started at http://127.3.240.193:37953/ using document root <none> and password file <none>
I20260927 23:07:26.284142 19598 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:26.284289 19598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:26.284633 19598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:07:26.287741 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "e48f1927e3eb4f958f61e22a21a43086"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "3aa6f4a404d99682b5ec63e5e3a3d593"
server_key_iv: "132e438eccf1cdf302131c97f6c52312"
server_key_version: "encryptionkey@0"
I20260927 23:07:26.288502 19598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "e48f1927e3eb4f958f61e22a21a43086"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "3aa6f4a404d99682b5ec63e5e3a3d593"
server_key_iv: "132e438eccf1cdf302131c97f6c52312"
server_key_version: "encryptionkey@0"
I20260927 23:07:26.293637 19598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260927 23:07:26.296998 19613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:26.298118 19598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:07:26.298282 19598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "e48f1927e3eb4f958f61e22a21a43086"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "3aa6f4a404d99682b5ec63e5e3a3d593"
server_key_iv: "132e438eccf1cdf302131c97f6c52312"
server_key_version: "encryptionkey@0"
I20260927 23:07:26.298509 19598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:26.331252 19598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:26.335032 19598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:26.335338 19598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:26.336263 19598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:26.337574 19598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:26.337672 19598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:26.337823 19598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:26.337894 19598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:26.371662 19598 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:45985
I20260927 23:07:26.371697 19725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:45985 every 8 connection(s)
I20260927 23:07:26.372864 19598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:07:26.377884 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 19598
I20260927 23:07:26.378120 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:07:26.378716 4035 external_mini_cluster.cc:1428] Setting key 108cde8e2ef3bca89fc649cfc989ffb9
Sep 27 23:07:26 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:26.392578 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:26.375385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:56249 (local address 127.3.240.254:45111)
0927 23:07:26.375809 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:26.375816 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:26.376772 (+ 956us) server_negotiation.cc:407] Connection header received
0927 23:07:26.378689 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:26.378694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:26.378779 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:26.378915 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:26.380749 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:26.381439 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:26.382914 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:26.383407 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:26.383544 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:26.386938 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:26.386943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:26.386946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:26.386977 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:26.389634 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:26.390337 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:26.390342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:26.390348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:26.390467 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:26.390892 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:26.390897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:26.390900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:26.391205 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:26.391352 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:26.392010 (+ 658us) server_negotiation.cc:299] Negotiation successful
0927 23:07:26.392231 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":197,"threads_started":1}
I20260927 23:07:26.393496 19729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:26.375976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:56249)
0927 23:07:26.376618 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:26.376656 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:26.378430 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:26.379180 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:26.379188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:26.379873 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:26.380593 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:26.380620 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:26.381593 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:26.381598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:26.382764 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:26.382775 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:26.383658 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:26.383668 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:26.383901 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:26.384609 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:26.384634 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:26.386720 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:26.389774 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:26.389783 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:26.389799 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:26.390208 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:26.390581 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:26.390585 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:26.390602 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:26.390776 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:26.391324 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:26.391331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:26.391650 (+ 319us) client_negotiation.cc:769] Sending connection context
0927 23:07:26.391900 (+ 250us) client_negotiation.cc:240] Negotiation successful
0927 23:07:26.392380 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":254,"threads_started":1}
I20260927 23:07:26.395035 19726 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45111
I20260927 23:07:26.395352 19726 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:26.396193 19726 heartbeater.cc:511] Master 127.3.240.254:45111 requested a full tablet report, sending...
I20260927 23:07:26.398507 19537 ts_manager.cc:194] Registered new tserver with Master: e48f1927e3eb4f958f61e22a21a43086 (127.3.240.193:45985)
I20260927 23:07:26.400003 19537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:56249
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:07:26.426100 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45111
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:26.744215 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:26.744565 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:26.744755 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:26.752036 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:26.752200 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:26.752447 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:07:26.762357 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:26.763698 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:26.764981 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:26.778119 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:26.778178 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:26.778208 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:26.778980 19732 server_base.cc:1034] running on GCE node
I20260927 23:07:26.779552 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:26.780373 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:26.781560 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790550446781529 us; error 35 us; skew 500 ppm
I20260927 23:07:26.781862 19732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:26 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:26.785058 19732 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:07:26.786393 19732 webserver.cc:460] Webserver started at http://127.3.240.194:35123/ using document root <none> and password file <none>
I20260927 23:07:26.787096 19732 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:26.787227 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:26.787516 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:07:26.790113 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "0d450d031d53494c9c97e7948c85f3a1"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "8160e73066aab1d8ccd3d7479903c02e"
server_key_iv: "2cf1e3f1bff5acdd8f9383dd28d442ff"
server_key_version: "encryptionkey@0"
I20260927 23:07:26.790817 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "0d450d031d53494c9c97e7948c85f3a1"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "8160e73066aab1d8ccd3d7479903c02e"
server_key_iv: "2cf1e3f1bff5acdd8f9383dd28d442ff"
server_key_version: "encryptionkey@0"
I20260927 23:07:26.795724 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260927 23:07:26.799189 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:26.800499 19732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:07:26.800674 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "0d450d031d53494c9c97e7948c85f3a1"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "8160e73066aab1d8ccd3d7479903c02e"
server_key_iv: "2cf1e3f1bff5acdd8f9383dd28d442ff"
server_key_version: "encryptionkey@0"
I20260927 23:07:26.800875 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:26.814513 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:26.819036 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:26.819341 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:26.820276 19732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:26.821615 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:26.821718 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:26.821884 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:26.821961 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:26.856431 19732 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:35303
I20260927 23:07:26.856456 19859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:35303 every 8 connection(s)
I20260927 23:07:26.857707 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:07:26.864761 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 19732
I20260927 23:07:26.864934 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:07:26.865345 4035 external_mini_cluster.cc:1428] Setting key ab4acd1a4c809bf2e6f9fd6db329ea04
Sep 27 23:07:26 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550446, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:26.878324 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:26.860437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:46915 (local address 127.3.240.254:45111)
0927 23:07:26.860655 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:26.860659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:26.861702 (+ 1043us) server_negotiation.cc:407] Connection header received
0927 23:07:26.863545 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:26.863550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:26.863640 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:26.863749 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:26.865317 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:26.866191 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:26.867825 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:26.868249 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:26.868379 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:26.872080 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:26.872085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:26.872088 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:26.872117 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:26.875271 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:26.875921 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:26.875929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:26.875935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:26.876098 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:26.876510 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:26.876517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:26.876523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:26.876847 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:26.876992 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:26.877610 (+ 618us) server_negotiation.cc:299] Negotiation successful
0927 23:07:26.877879 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260927 23:07:26.878865 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:26.860826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:46915)
0927 23:07:26.861513 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:26.861566 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:26.863288 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:26.863979 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:26.863987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:26.864476 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:26.865156 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:26.865176 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:26.866335 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:26.866340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:26.867686 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:26.867697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:26.868456 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:26.868462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:26.868687 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:26.869340 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:26.869360 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:26.871869 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:26.875387 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:26.875393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:26.875404 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:26.875795 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:26.876196 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:26.876199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:26.876202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:26.876375 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:26.876950 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:26.876955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:26.877200 (+ 245us) client_negotiation.cc:769] Sending connection context
0927 23:07:26.877430 (+ 230us) client_negotiation.cc:240] Negotiation successful
0927 23:07:26.877808 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":265,"threads_started":1}
I20260927 23:07:26.880261 19860 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45111
I20260927 23:07:26.880565 19860 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:26.881210 19860 heartbeater.cc:511] Master 127.3.240.254:45111 requested a full tablet report, sending...
I20260927 23:07:26.882601 19537 ts_manager.cc:194] Registered new tserver with Master: 0d450d031d53494c9c97e7948c85f3a1 (127.3.240.194:35303)
I20260927 23:07:26.883419 19537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:46915
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:07:26.914299 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45111
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:27.235154 19865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:27.235504 19865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:27.235690 19865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:27.243090 19865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:27.243243 19865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:27.243430 19865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:07:27.253064 19865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:27.254379 19865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:27.255716 19865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:27.269052 19870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:27.269178 19871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:27.269138 19873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:27.270293 19865 server_base.cc:1034] running on GCE node
I20260927 23:07:27.270852 19865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:27.271821 19865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:27.273030 19865 hybrid_clock.cc:648] HybridClock initialized: now 1790550447272969 us; error 54 us; skew 500 ppm
I20260927 23:07:27.273312 19865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:27 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:27.276309 19865 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:07:27.277710 19865 webserver.cc:460] Webserver started at http://127.3.240.195:36641/ using document root <none> and password file <none>
I20260927 23:07:27.278342 19865 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:27.278436 19865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:27.278687 19865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:07:27.281626 19865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "057416415fab4337a9a71718ecad7fac"
format_stamp: "Formatted at 2026-09-27 23:07:27 on dist-test-slave-lmfp"
server_key: "5db8e2c3cbb9f9ee5c0e0f93e8c1dfdb"
server_key_iv: "d97335439690bf939030ca5f2aa951ed"
server_key_version: "encryptionkey@0"
I20260927 23:07:27.282323 19865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "057416415fab4337a9a71718ecad7fac"
format_stamp: "Formatted at 2026-09-27 23:07:27 on dist-test-slave-lmfp"
server_key: "5db8e2c3cbb9f9ee5c0e0f93e8c1dfdb"
server_key_iv: "d97335439690bf939030ca5f2aa951ed"
server_key_version: "encryptionkey@0"
I20260927 23:07:27.287478 19865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 23:07:27.290762 19880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:27.292106 19865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:07:27.292274 19865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "057416415fab4337a9a71718ecad7fac"
format_stamp: "Formatted at 2026-09-27 23:07:27 on dist-test-slave-lmfp"
server_key: "5db8e2c3cbb9f9ee5c0e0f93e8c1dfdb"
server_key_iv: "d97335439690bf939030ca5f2aa951ed"
server_key_version: "encryptionkey@0"
I20260927 23:07:27.292456 19865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:27.317252 19865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:27.321297 19865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:27.321592 19865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:27.322618 19865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:27.323973 19865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:27.324085 19865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:27.324229 19865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:27.324296 19865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:27.357939 19865 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:41999
I20260927 23:07:27.358055 19992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:41999 every 8 connection(s)
I20260927 23:07:27.359469 19865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:07:27.365549 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 19865
I20260927 23:07:27.365748 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:07:27.366178 4035 external_mini_cluster.cc:1428] Setting key 7792c8e9e193d3c4762425b9c2ebf5f1
Sep 27 23:07:27 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550447, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:27.380259 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.362314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:48031 (local address 127.3.240.254:45111)
0927 23:07:27.362497 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.362502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.363711 (+ 1209us) server_negotiation.cc:407] Connection header received
0927 23:07:27.365519 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.365523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.365598 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.365730 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:27.367408 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.368424 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.370032 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.370401 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.370496 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.373972 (+ 3476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:27.373979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:27.373983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:27.374021 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:27.377297 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:27.377926 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:27.377930 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:27.377932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:27.378057 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:27.378394 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:27.378397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:27.378399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:27.378653 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:27.378765 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.379658 (+ 893us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.379881 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:07:27.381223 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.362800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:48031)
0927 23:07:27.363553 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:27.363595 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:27.365211 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:27.366105 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:27.366113 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:27.366669 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:27.367272 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.367292 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.368580 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.368584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.369903 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.369915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.370584 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.370589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.370790 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.371532 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:27.371552 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:27.373775 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:27.377416 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:27.377422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:27.377431 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:27.377815 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:27.378145 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:27.378148 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:27.378150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:27.378300 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:27.378801 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:27.378807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:27.379224 (+ 417us) client_negotiation.cc:769] Sending connection context
0927 23:07:27.379524 (+ 300us) client_negotiation.cc:240] Negotiation successful
0927 23:07:27.380029 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":347,"threads_started":1}
I20260927 23:07:27.382620 19993 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45111
I20260927 23:07:27.382942 19993 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:27.383651 19993 heartbeater.cc:511] Master 127.3.240.254:45111 requested a full tablet report, sending...
I20260927 23:07:27.384991 19537 ts_manager.cc:194] Registered new tserver with Master: 057416415fab4337a9a71718ecad7fac (127.3.240.195:41999)
I20260927 23:07:27.385778 19537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:48031
I20260927 23:07:27.393858 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:07:27.402853 19726 heartbeater.cc:503] Master 127.3.240.254:45111 was elected leader, sending a full tablet report...
I20260927 23:07:27.411980 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.398755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49724 (local address 127.3.240.254:45111)
0927 23:07:27.398955 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.398964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.399411 (+ 447us) server_negotiation.cc:407] Connection header received
0927 23:07:27.399604 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.399609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.399678 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.399780 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:27.401238 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.401929 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.403386 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.403869 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.403966 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.406711 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:27.406715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:27.406719 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:27.406750 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:27.409560 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:27.410240 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:27.410256 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:27.410258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:27.410382 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:27.410706 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:27.410709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:27.410711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:27.411051 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:27.411248 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.411494 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.411655 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:07:27.415740 19537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:07:27.422500 19537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:07:27.457172 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.434269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.3.240.194:35303)
0927 23:07:27.441352 (+ 7083us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.441366 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.445782 (+ 4416us) server_negotiation.cc:407] Connection header received
0927 23:07:27.447291 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.447304 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.447599 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.447770 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:27.449476 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.451309 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.454406 (+ 3097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.455965 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.456110 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.456320 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.456670 (+ 350us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.456848 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6856,"thread_start_us":6735,"threads_started":1}
I20260927 23:07:27.457250 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.434212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:35303 (local address 127.0.0.1:48632)
0927 23:07:27.445560 (+ 11348us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:27.445612 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:27.446705 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:27.448112 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:27.448119 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:27.448145 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:27.448570 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.448588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.451628 (+ 3040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.451634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.454218 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.454249 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.456201 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.456208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.456417 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.456472 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 23:07:27.456607 (+ 135us) client_negotiation.cc:240] Negotiation successful
0927 23:07:27.456779 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11039,"mutex_wait_us":31,"thread_start_us":18249,"threads_started":1}
I20260927 23:07:27.460142 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.434252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:41999 (local address 127.0.0.1:49974)
0927 23:07:27.450066 (+ 15814us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:27.450085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:27.450219 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:27.452770 (+ 2551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:27.452774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:27.452802 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:27.453308 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.453321 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.455024 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.455032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.457389 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.457405 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.459265 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.459271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.459429 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.459459 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 23:07:27.459557 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 23:07:27.459707 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15699,"mutex_wait_us":231,"thread_start_us":10892,"threads_started":1}
I20260927 23:07:27.460183 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.434896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49974 (local address 127.3.240.195:41999)
0927 23:07:27.451857 (+ 16961us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.451869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.451912 (+ 43us) server_negotiation.cc:407] Connection header received
0927 23:07:27.452020 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.452031 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.452324 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.452796 (+ 472us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:27.453535 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.454857 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.457565 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.459060 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.459213 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.459432 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.459626 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.459808 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16760,"thread_start_us":16588,"threads_started":1}
I20260927 23:07:27.461382 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.435015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33310 (local address 127.3.240.193:45985)
0927 23:07:27.452470 (+ 17455us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.452481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.452510 (+ 29us) server_negotiation.cc:407] Connection header received
0927 23:07:27.452723 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.452732 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.452993 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.453188 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:27.453848 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.455276 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.457751 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.459119 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.459233 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.459432 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.460994 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.461161 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17259,"thread_start_us":17495,"threads_started":1}
I20260927 23:07:27.461882 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.434631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45985 (local address 127.0.0.1:33310)
0927 23:07:27.452385 (+ 17754us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:27.452403 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:27.452518 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:27.453219 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:27.453223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:27.453258 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:27.453687 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.453696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.455459 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.455464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.457573 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.457588 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.459597 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.459603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.459800 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.459830 (+ 30us) client_negotiation.cc:769] Sending connection context
0927 23:07:27.459934 (+ 104us) client_negotiation.cc:240] Negotiation successful
0927 23:07:27.460611 (+ 677us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17596,"mutex_wait_us":1,"thread_start_us":16761,"threads_started":1}
I20260927 23:07:27.461869 19795 tablet_service.cc:1467] Processing CreateTablet for tablet 060fc2f42e314f8f830b65573c9611b2 (DEFAULT_TABLE table=test-table [id=133e04bbb6c14b3284cae7ff34571180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:27.463980 19928 tablet_service.cc:1467] Processing CreateTablet for tablet 060fc2f42e314f8f830b65573c9611b2 (DEFAULT_TABLE table=test-table [id=133e04bbb6c14b3284cae7ff34571180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:27.465134 19928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060fc2f42e314f8f830b65573c9611b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:27.465197 19795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060fc2f42e314f8f830b65573c9611b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:27.465055 19661 tablet_service.cc:1467] Processing CreateTablet for tablet 060fc2f42e314f8f830b65573c9611b2 (DEFAULT_TABLE table=test-table [id=133e04bbb6c14b3284cae7ff34571180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:27.466392 19661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 060fc2f42e314f8f830b65573c9611b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:27.476320 20011 tablet_bootstrap.cc:492] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Bootstrap starting.
I20260927 23:07:27.479678 20011 tablet_bootstrap.cc:654] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:27.480429 20012 tablet_bootstrap.cc:492] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Bootstrap starting.
I20260927 23:07:27.481072 20011 log.cc:824] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:27.481386 20013 tablet_bootstrap.cc:492] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Bootstrap starting.
I20260927 23:07:27.484334 20013 tablet_bootstrap.cc:654] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:27.484464 20012 tablet_bootstrap.cc:654] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:27.485554 20013 log.cc:824] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:27.485627 20012 log.cc:824] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:27.487426 20011 tablet_bootstrap.cc:492] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: No bootstrap required, opened a new log
I20260927 23:07:27.487742 20011 ts_tablet_manager.cc:1397] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260927 23:07:27.489362 20012 tablet_bootstrap.cc:492] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: No bootstrap required, opened a new log
I20260927 23:07:27.489634 20013 tablet_bootstrap.cc:492] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: No bootstrap required, opened a new log
I20260927 23:07:27.489635 20012 ts_tablet_manager.cc:1397] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260927 23:07:27.489878 20013 ts_tablet_manager.cc:1397] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 23:07:27.498064 20012 raft_consensus.cc:348] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.498517 20012 raft_consensus.cc:374] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:27.498617 20012 raft_consensus.cc:729] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d450d031d53494c9c97e7948c85f3a1, State: Initialized, Role: FOLLOWER
I20260927 23:07:27.498744 20011 raft_consensus.cc:348] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.499200 20011 raft_consensus.cc:374] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:27.499292 20011 raft_consensus.cc:729] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057416415fab4337a9a71718ecad7fac, State: Initialized, Role: FOLLOWER
I20260927 23:07:27.499341 20012 consensus_queue.cc:260] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.500031 20011 consensus_queue.cc:260] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.500808 20013 raft_consensus.cc:348] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.501328 20013 raft_consensus.cc:374] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:27.501358 19993 heartbeater.cc:503] Master 127.3.240.254:45111 was elected leader, sending a full tablet report...
I20260927 23:07:27.501536 20013 raft_consensus.cc:729] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48f1927e3eb4f958f61e22a21a43086, State: Initialized, Role: FOLLOWER
I20260927 23:07:27.502017 20011 ts_tablet_manager.cc:1428] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260927 23:07:27.502303 20013 consensus_queue.cc:260] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.502964 19860 heartbeater.cc:503] Master 127.3.240.254:45111 was elected leader, sending a full tablet report...
I20260927 23:07:27.503736 20012 ts_tablet_manager.cc:1428] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260927 23:07:27.505514 20013 ts_tablet_manager.cc:1428] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
W20260927 23:07:27.609987 19861 tablet.cc:2369] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:07:27.611404 19994 tablet.cc:2369] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:07:27.625219 19727 tablet.cc:2369] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:07:27.651998 20018 raft_consensus.cc:482] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:27.652314 20018 raft_consensus.cc:504] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.653985 20018 leader_election.cc:288] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d450d031d53494c9c97e7948c85f3a1 (127.3.240.194:35303), e48f1927e3eb4f958f61e22a21a43086 (127.3.240.193:45985)
I20260927 23:07:27.659433 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.654608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:46131 (local address 127.3.240.193:45985)
0927 23:07:27.654800 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.654804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.654819 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:07:27.654878 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.654881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.654935 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.655131 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:27.655540 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.656232 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.657772 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.658532 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.658692 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.658785 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.659090 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.659209 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 23:07:27.659427 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.654295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45985 (local address 127.3.240.195:46131)
0927 23:07:27.654505 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:27.654526 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:27.654652 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:27.655155 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:27.655160 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:27.655195 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:27.655390 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.655399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.656363 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.656369 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.657584 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.657594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.658679 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.658684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.658881 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.658905 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 23:07:27.659026 (+ 121us) client_negotiation.cc:240] Negotiation successful
0927 23:07:27.659155 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 23:07:27.660465 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.654322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:35303 (local address 127.3.240.195:33219)
0927 23:07:27.655451 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:27.655465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:27.655594 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:27.655911 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:27.655915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:27.655936 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:27.656252 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.656260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.657236 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.657247 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.658542 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:27.658552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.659860 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.659866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:27.660024 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.660058 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 23:07:27.660145 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 23:07:27.660222 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1027,"mutex_wait_us":356,"thread_start_us":407,"threads_started":1}
I20260927 23:07:27.660597 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.654423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:33219 (local address 127.3.240.194:35303)
0927 23:07:27.654609 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.654614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.655554 (+ 940us) server_negotiation.cc:407] Connection header received
0927 23:07:27.655735 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.655741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.655808 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.655899 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:27.656400 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.657082 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.658692 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.659643 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.659784 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.659867 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.660203 (+ 336us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.660332 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 23:07:27.660638 19681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060fc2f42e314f8f830b65573c9611b2" candidate_uuid: "057416415fab4337a9a71718ecad7fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48f1927e3eb4f958f61e22a21a43086" is_pre_election: true
I20260927 23:07:27.661191 19681 raft_consensus.cc:2454] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 057416415fab4337a9a71718ecad7fac in term 0.
I20260927 23:07:27.661751 19815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060fc2f42e314f8f830b65573c9611b2" candidate_uuid: "057416415fab4337a9a71718ecad7fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d450d031d53494c9c97e7948c85f3a1" is_pre_election: true
I20260927 23:07:27.661911 19884 leader_election.cc:302] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057416415fab4337a9a71718ecad7fac, e48f1927e3eb4f958f61e22a21a43086; no voters:
I20260927 23:07:27.662317 20018 raft_consensus.cc:2781] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:07:27.662263 19815 raft_consensus.cc:2454] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 057416415fab4337a9a71718ecad7fac in term 0.
I20260927 23:07:27.662463 20018 raft_consensus.cc:482] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:07:27.662569 20018 raft_consensus.cc:3037] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:27.665009 20018 raft_consensus.cc:504] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.665555 20018 leader_election.cc:288] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [CANDIDATE]: Term 1 election: Requested vote from peers 0d450d031d53494c9c97e7948c85f3a1 (127.3.240.194:35303), e48f1927e3eb4f958f61e22a21a43086 (127.3.240.193:45985)
I20260927 23:07:27.665969 19815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060fc2f42e314f8f830b65573c9611b2" candidate_uuid: "057416415fab4337a9a71718ecad7fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d450d031d53494c9c97e7948c85f3a1"
I20260927 23:07:27.666004 19681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "060fc2f42e314f8f830b65573c9611b2" candidate_uuid: "057416415fab4337a9a71718ecad7fac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48f1927e3eb4f958f61e22a21a43086"
I20260927 23:07:27.666173 19815 raft_consensus.cc:3037] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:27.666211 19681 raft_consensus.cc:3037] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:27.668419 19681 raft_consensus.cc:2454] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 057416415fab4337a9a71718ecad7fac in term 1.
I20260927 23:07:27.668422 19815 raft_consensus.cc:2454] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 057416415fab4337a9a71718ecad7fac in term 1.
I20260927 23:07:27.668787 19884 leader_election.cc:302] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057416415fab4337a9a71718ecad7fac, e48f1927e3eb4f958f61e22a21a43086; no voters:
I20260927 23:07:27.669080 20018 raft_consensus.cc:2781] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:07:27.669529 20018 raft_consensus.cc:686] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 1 LEADER]: Becoming Leader. State: Replica: 057416415fab4337a9a71718ecad7fac, State: Running, Role: LEADER
I20260927 23:07:27.669950 20018 consensus_queue.cc:237] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:27.674297 19537 catalog_manager.cc:5495] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac reported cstate change: term changed from 0 to 1, leader changed from <none> to 057416415fab4337a9a71718ecad7fac (127.3.240.195). New cstate: current_term: 1 leader_uuid: "057416415fab4337a9a71718ecad7fac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20260927 23:07:27.691520 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.685608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49986 (local address 127.3.240.195:41999)
0927 23:07:27.685888 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.685897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.686042 (+ 145us) server_negotiation.cc:407] Connection header received
0927 23:07:27.686308 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.686312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.686387 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.686557 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:07:27.686964 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.687724 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.689303 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.689714 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.689863 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.690395 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:07:27.690861 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:07:27.691095 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.691151 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.691232 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 23:07:27.699693 19815 raft_consensus.cc:1261] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 FOLLOWER]: Refusing update from remote peer 057416415fab4337a9a71718ecad7fac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:07:27.699707 19681 raft_consensus.cc:1261] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 1 FOLLOWER]: Refusing update from remote peer 057416415fab4337a9a71718ecad7fac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:07:27.700637 20018 consensus_queue.cc:1035] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:07:27.700966 20021 consensus_queue.cc:1035] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:07:27.730865 19536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:07:27.731199 19536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:07:27.734452 19536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Sending DeleteTablet for 3 replicas of tablet 060fc2f42e314f8f830b65573c9611b2
I20260927 23:07:27.735641 19795 tablet_service.cc:1514] Processing DeleteTablet for tablet 060fc2f42e314f8f830b65573c9611b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:27 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:48632
I20260927 23:07:27.735673 19928 tablet_service.cc:1514] Processing DeleteTablet for tablet 060fc2f42e314f8f830b65573c9611b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:27 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:49974
I20260927 23:07:27.736194 19661 tablet_service.cc:1514] Processing DeleteTablet for tablet 060fc2f42e314f8f830b65573c9611b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:27 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:33310
I20260927 23:07:27.738065 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 19598
I20260927 23:07:27.747646 20036 tablet_replica.cc:330] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: stopping tablet replica
I20260927 23:07:27.749357 20036 raft_consensus.cc:2229] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:07:27.750443 20036 raft_consensus.cc:2258] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:27.751278 20035 tablet_replica.cc:330] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: stopping tablet replica
I20260927 23:07:27.751854 20034 tablet_replica.cc:330] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: stopping tablet replica
I20260927 23:07:27.752282 20034 raft_consensus.cc:2229] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:07:27.752954 20034 raft_consensus.cc:2258] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:27.754544 20036 ts_tablet_manager.cc:1905] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 27 23:07:27 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:07:27.761515 20036 ts_tablet_manager.cc:1918] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:27.761675 20036 log.cc:1196] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/060fc2f42e314f8f830b65573c9611b2
I20260927 23:07:27.762243 20036 ts_tablet_manager.cc:1939] T 060fc2f42e314f8f830b65573c9611b2 P e48f1927e3eb4f958f61e22a21a43086: Deleting consensus metadata
I20260927 23:07:27.763561 20035 raft_consensus.cc:2229] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:07:27.764031 20035 raft_consensus.cc:2258] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:27.766016 20035 ts_tablet_manager.cc:1905] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:27.766390 20034 ts_tablet_manager.cc:1905] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:27.772583 20035 ts_tablet_manager.cc:1918] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:27.772732 20035 log.cc:1196] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/060fc2f42e314f8f830b65573c9611b2
I20260927 23:07:27.773175 20035 ts_tablet_manager.cc:1939] T 060fc2f42e314f8f830b65573c9611b2 P 0d450d031d53494c9c97e7948c85f3a1: Deleting consensus metadata
I20260927 23:07:27.774945 19523 catalog_manager.cc:4841] TS 0d450d031d53494c9c97e7948c85f3a1 (127.3.240.194:35303): tablet 060fc2f42e314f8f830b65573c9611b2 (table test-table [id=133e04bbb6c14b3284cae7ff34571180]) successfully deleted
I20260927 23:07:27.775799 19524 catalog_manager.cc:4841] TS e48f1927e3eb4f958f61e22a21a43086 (127.3.240.193:45985): tablet 060fc2f42e314f8f830b65573c9611b2 (table test-table [id=133e04bbb6c14b3284cae7ff34571180]) successfully deleted
I20260927 23:07:27.777251 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.739381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33320 (local address 127.3.240.193:45985)
0927 23:07:27.739603 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.739611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.745180 (+ 5569us) server_negotiation.cc:407] Connection header received
0927 23:07:27.745408 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.745414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.745518 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.745635 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:27.749302 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.750494 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.752751 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.753367 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.753466 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.759585 (+ 6119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:27.759604 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:27.759621 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:27.759676 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:27.764834 (+ 5158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:27.774295 (+ 9461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:27.774305 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:27.774309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:27.774510 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:27.775176 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:27.775182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:27.775185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:27.775841 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:27.776161 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.776644 (+ 483us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.776866 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 23:07:27.777531 20034 ts_tablet_manager.cc:1918] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:27.777752 20034 log.cc:1196] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/060fc2f42e314f8f830b65573c9611b2
I20260927 23:07:27.778599 20034 ts_tablet_manager.cc:1939] T 060fc2f42e314f8f830b65573c9611b2 P 057416415fab4337a9a71718ecad7fac: Deleting consensus metadata
I20260927 23:07:27.778628 19721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:27.780586 19522 catalog_manager.cc:4841] TS 057416415fab4337a9a71718ecad7fac (127.3.240.195:41999): tablet 060fc2f42e314f8f830b65573c9611b2 (table test-table [id=133e04bbb6c14b3284cae7ff34571180]) successfully deleted
I20260927 23:07:27.887290 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 19598
I20260927 23:07:27.905655 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 19732
Sep 27 23:07:27 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:07:27.922663 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:27.906406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.3.240.194:35303)
0927 23:07:27.906606 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:27.906612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:27.906683 (+ 71us) server_negotiation.cc:407] Connection header received
0927 23:07:27.906910 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:27.906914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:27.906978 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:27.907136 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:27.908530 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.909251 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.910592 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:27.910963 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:27.911156 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:27.915810 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:27.915830 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:27.915863 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:27.915911 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:27.919661 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:27.920351 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:27.920361 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:27.920366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:27.920535 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:27.920922 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:27.920927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:27.920931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:27.921457 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:27.921764 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:27.922013 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 23:07:27.922240 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 23:07:27.923442 19855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:28.010151 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 19732
I20260927 23:07:28.027055 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 19865
Sep 27 23:07:28 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:07:28.046300 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:28.028006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49994 (local address 127.3.240.195:41999)
0927 23:07:28.028169 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:28.028173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:28.028237 (+ 64us) server_negotiation.cc:407] Connection header received
0927 23:07:28.028448 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:28.028452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:28.028514 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:28.028607 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:28.030632 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:28.031613 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:28.033500 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:28.033926 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:28.034014 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:28.039871 (+ 5857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:28.039883 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:28.039896 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:28.039937 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:28.043246 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:28.044134 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:28.044140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:28.044142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:28.044295 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:28.044747 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:28.044750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:28.044752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:28.045094 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:28.045305 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:28.045755 (+ 450us) server_negotiation.cc:299] Negotiation successful
0927 23:07:28.045911 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:07:28.047206 19988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:28.167483 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 19865
I20260927 23:07:28.186596 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 19505
I20260927 23:07:28.187768 19567 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:28.295593 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 19505
I20260927 23:07:48.317442 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45111
--webserver_interface=127.3.240.254
--webserver_port=35639
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:07:48.633620 20042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:48.633911 20042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:48.634063 20042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:48.640913 20042 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 23:07:48.641038 20042 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 23:07:48.641129 20042 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 23:07:48.641219 20042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:07:48.641316 20042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:48.641427 20042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:48.641505 20042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:07:48.641601 20042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:07:48.650808 20042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=35639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:48.652027 20042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:48.653290 20042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:48.674835 20048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:48.674770 20051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:48.674813 20049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:48.675226 20042 server_base.cc:1034] running on GCE node
I20260927 23:07:48.676218 20042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:48.677663 20042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:48.678915 20042 hybrid_clock.cc:648] HybridClock initialized: now 1790550468678844 us; error 48 us; skew 500 ppm
I20260927 23:07:48.679288 20042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:48 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:48.686208 20042 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:07:48.687929 20042 webserver.cc:460] Webserver started at http://127.3.240.254:35639/ using document root <none> and password file <none>
I20260927 23:07:48.688618 20042 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:48.688714 20042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:48.694346 20042 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:07:48.697420 20058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:48.698755 20042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:07:48.698977 20042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "14c7b77c729849479fcbc2041e851bbf"
format_stamp: "Formatted at 2026-09-27 23:07:25 on dist-test-slave-lmfp"
server_key: "ff853b49d2514f15f24971097c29ee6c"
server_key_iv: "920ddc82cd9d8b458cc8764f3cecf915"
server_key_version: "encryptionkey@0"
I20260927 23:07:48.700022 20042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:48.714944 20042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:48.718976 20042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:48.719282 20042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:48.737704 20042 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:45111
I20260927 23:07:48.737735 20109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:45111 every 8 connection(s)
I20260927 23:07:48.739101 20042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:07:48.745545 20110 sys_catalog.cc:263] Verifying existing consensus state
I20260927 23:07:48.748667 20110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Bootstrap starting.
I20260927 23:07:48.748962 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 20042
I20260927 23:07:48.749738 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:45985
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:45111
--webserver_port=37953
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:07:48.767971 20110 log.cc:824] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:48.779608 20110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:07:48.780159 20110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Bootstrap complete.
I20260927 23:07:48.788810 20110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:07:48.789561 20110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c7b77c729849479fcbc2041e851bbf, State: Initialized, Role: FOLLOWER
I20260927 23:07:48.790187 20110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:07:48.790409 20110 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:07:48.790534 20110 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:07:48.790714 20110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 1 FOLLOWER]: Advancing to term 2
I20260927 23:07:48.794458 20110 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:07:48.794972 20110 leader_election.cc:302] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14c7b77c729849479fcbc2041e851bbf; no voters:
I20260927 23:07:48.795718 20110 leader_election.cc:288] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260927 23:07:48.795928 20114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 2 FOLLOWER]: Leader election won for term 2
I20260927 23:07:48.797608 20114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 2 LEADER]: Becoming Leader. State: Replica: 14c7b77c729849479fcbc2041e851bbf, State: Running, Role: LEADER
I20260927 23:07:48.798195 20110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:07:48.798137 20114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:07:48.800720 20115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "14c7b77c729849479fcbc2041e851bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } } }
I20260927 23:07:48.801116 20115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:48.801673 20116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14c7b77c729849479fcbc2041e851bbf. Latest consensus state: current_term: 2 leader_uuid: "14c7b77c729849479fcbc2041e851bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } } }
I20260927 23:07:48.802019 20116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: This master's current role is: LEADER
I20260927 23:07:48.804356 20121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:07:48.813170 20121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:07:48.814913 20121 catalog_manager.cc:1259] Loaded cluster ID: f9df7fb6922640eebaff7a77012d082c
I20260927 23:07:48.815038 20121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:07:48.820032 20121 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:07:48.821951 20121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Loaded TSK: 0
I20260927 23:07:48.850289 20131 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 23:07:48.862526 20121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Generated new TSK 1
I20260927 23:07:48.862872 20121 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 23:07:49.099553 20112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:49.099836 20112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:49.099990 20112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:49.110018 20112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:49.110158 20112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:49.110379 20112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:07:49.120813 20112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:45985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=37953
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:49.122164 20112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:49.123505 20112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:49.137949 20137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:49.138399 20140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:49.138712 20138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:49.139613 20112 server_base.cc:1034] running on GCE node
I20260927 23:07:49.140317 20112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:49.141309 20112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:49.142550 20112 hybrid_clock.cc:648] HybridClock initialized: now 1790550469142486 us; error 57 us; skew 500 ppm
I20260927 23:07:49.142875 20112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:49 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:49.146066 20112 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:07:49.147653 20112 webserver.cc:460] Webserver started at http://127.3.240.193:37953/ using document root <none> and password file <none>
I20260927 23:07:49.148329 20112 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:49.148463 20112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:49.153833 20112 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 23:07:49.156577 20147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:49.157898 20112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260927 23:07:49.158083 20112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "e48f1927e3eb4f958f61e22a21a43086"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "3aa6f4a404d99682b5ec63e5e3a3d593"
server_key_iv: "132e438eccf1cdf302131c97f6c52312"
server_key_version: "encryptionkey@0"
I20260927 23:07:49.159027 20112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:49.184297 20112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:49.188333 20112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:49.188594 20112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:49.189568 20112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:49.190939 20112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:49.191080 20112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260927 23:07:49.191198 20112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:49.191277 20112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:49.224056 20112 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:45985
I20260927 23:07:49.224138 20259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:45985 every 8 connection(s)
I20260927 23:07:49.225461 20112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:07:49.232522 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 20112
I20260927 23:07:49.233312 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:35303
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:45111
--webserver_port=35123
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:07:49 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:49.250874 20263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:49.228709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:47659)
0927 23:07:49.229558 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:49.229602 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:49.231366 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:49.232610 (+ 1244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:49.232622 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:49.233348 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:49.234067 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:49.234093 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:49.235648 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:49.235655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:49.237790 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:49.237808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:49.238760 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:49.238769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:49.239084 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:49.239784 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:49.239817 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:49.242495 (+ 2678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:49.247164 (+ 4669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:49.247170 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:49.247181 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:49.247564 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:49.248049 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:49.248052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:49.248054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:49.248221 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:49.249127 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:49.249131 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:49.249393 (+ 262us) client_negotiation.cc:769] Sending connection context
0927 23:07:49.249604 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 23:07:49.249935 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":369,"threads_started":1}
I20260927 23:07:49.251574 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:49.228420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:47659 (local address 127.3.240.254:45111)
0927 23:07:49.228965 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:49.228977 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:49.229726 (+ 749us) server_negotiation.cc:407] Connection header received
0927 23:07:49.231810 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:49.231822 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:49.232121 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:49.232382 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:49.234255 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:49.235473 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:49.237940 (+ 2467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:49.238550 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:49.238670 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:49.242765 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:49.242777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:49.242797 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:49.242856 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:49.246987 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:49.247694 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:49.247706 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:49.247716 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:49.247925 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:49.248334 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:49.248342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:49.248348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:49.248997 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:49.249193 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:49.249847 (+ 654us) server_negotiation.cc:299] Negotiation successful
0927 23:07:49.250278 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260927 23:07:49.254227 20260 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45111
I20260927 23:07:49.254523 20260 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:49.255256 20260 heartbeater.cc:511] Master 127.3.240.254:45111 requested a full tablet report, sending...
I20260927 23:07:49.257311 20075 ts_manager.cc:194] Registered new tserver with Master: e48f1927e3eb4f958f61e22a21a43086 (127.3.240.193:45985)
I20260927 23:07:49.258713 20075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:47659
W20260927 23:07:49.544793 20264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:49.545085 20264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:49.545252 20264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:49.552106 20264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:49.552229 20264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:49.552373 20264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:07:49.561417 20264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:35303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=35123
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:49.562660 20264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:49.563989 20264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:49.576274 20270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:49.576398 20269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:49.576327 20272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:49.577088 20264 server_base.cc:1034] running on GCE node
I20260927 23:07:49.577723 20264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:49.578591 20264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:49.579805 20264 hybrid_clock.cc:648] HybridClock initialized: now 1790550469579771 us; error 56 us; skew 500 ppm
I20260927 23:07:49.580137 20264 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:49 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:49.582973 20264 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:07:49.584332 20264 webserver.cc:460] Webserver started at http://127.3.240.194:35123/ using document root <none> and password file <none>
I20260927 23:07:49.584915 20264 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:49.585016 20264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:49.590126 20264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260927 23:07:49.592564 20279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:49.593724 20264 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:07:49.593899 20264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "0d450d031d53494c9c97e7948c85f3a1"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "8160e73066aab1d8ccd3d7479903c02e"
server_key_iv: "2cf1e3f1bff5acdd8f9383dd28d442ff"
server_key_version: "encryptionkey@0"
I20260927 23:07:49.594831 20264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:49.617486 20264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:49.621168 20264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:49.621423 20264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:49.622381 20264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:49.623695 20264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:49.623806 20264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:49.623934 20264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:49.624027 20264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:49.656699 20264 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:35303
I20260927 23:07:49.656780 20391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:35303 every 8 connection(s)
I20260927 23:07:49.658116 20264 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:07:49.658739 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 20264
I20260927 23:07:49.659590 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:41999
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:45111
--webserver_port=36641
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 27 23:07:49 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:49.678648 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:49.660856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:46813 (local address 127.3.240.254:45111)
0927 23:07:49.661069 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:49.661076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:49.662393 (+ 1317us) server_negotiation.cc:407] Connection header received
0927 23:07:49.665200 (+ 2807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:49.665204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:49.665269 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:49.665361 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:49.667409 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:49.668545 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:49.669983 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:49.670376 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:49.670504 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:49.673525 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:49.673530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:49.673533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:49.673561 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:49.675869 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:49.676515 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:49.676521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:49.676525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:49.676670 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:49.677054 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:49.677059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:49.677061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:49.677330 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:49.677481 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:49.678087 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 23:07:49.678304 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 23:07:49.679612 20394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:49.661342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:46813)
0927 23:07:49.662193 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:49.662245 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:49.664930 (+ 2685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:49.665640 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:49.665652 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:49.666358 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:49.667210 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:49.667237 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:49.668702 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:49.668706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:49.669837 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:49.669847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:49.670583 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:49.670589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:49.670785 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:49.671428 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:49.671448 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:49.673293 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:49.676010 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:49.676018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:49.676031 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:49.676392 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:49.676774 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:49.676778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:49.676782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:49.676947 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:49.677457 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:49.677464 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:49.677761 (+ 297us) client_negotiation.cc:769] Sending connection context
0927 23:07:49.677997 (+ 236us) client_negotiation.cc:240] Negotiation successful
0927 23:07:49.678476 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":291,"threads_started":1}
I20260927 23:07:49.681208 20392 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45111
I20260927 23:07:49.681490 20392 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:49.682143 20392 heartbeater.cc:511] Master 127.3.240.254:45111 requested a full tablet report, sending...
I20260927 23:07:49.683679 20075 ts_manager.cc:194] Registered new tserver with Master: 0d450d031d53494c9c97e7948c85f3a1 (127.3.240.194:35303)
I20260927 23:07:49.684695 20075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:46813
W20260927 23:07:49.970991 20395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:07:49.971323 20395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:07:49.971447 20395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:07:49.978101 20395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:07:49.978184 20395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:07:49.978279 20395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:07:49.986953 20395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:41999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=36641
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:07:49.987998 20395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:07:49.989086 20395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:07:50.001861 20400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:50.001825 20401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:07:50.001792 20403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:07:50.002574 20395 server_base.cc:1034] running on GCE node
I20260927 23:07:50.003304 20395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:07:50.004292 20395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:07:50.005517 20395 hybrid_clock.cc:648] HybridClock initialized: now 1790550470005475 us; error 47 us; skew 500 ppm
I20260927 23:07:50.005841 20395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:07:50 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:07:50.009302 20395 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:07:50.010835 20395 webserver.cc:460] Webserver started at http://127.3.240.195:36641/ using document root <none> and password file <none>
I20260927 23:07:50.011574 20395 fs_manager.cc:359] Metadata directory not provided
I20260927 23:07:50.011747 20395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:07:50.016783 20395 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 23:07:50.019426 20410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:50.020628 20395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:07:50.020812 20395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "057416415fab4337a9a71718ecad7fac"
format_stamp: "Formatted at 2026-09-27 23:07:27 on dist-test-slave-lmfp"
server_key: "5db8e2c3cbb9f9ee5c0e0f93e8c1dfdb"
server_key_iv: "d97335439690bf939030ca5f2aa951ed"
server_key_version: "encryptionkey@0"
I20260927 23:07:50.021791 20395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:07:50.035663 20395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:07:50.039909 20395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:07:50.040192 20395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:07:50.041144 20395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:07:50.042495 20395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:07:50.042636 20395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:50.042783 20395 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:07:50.042874 20395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:07:50.076112 20395 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:41999
I20260927 23:07:50.076164 20522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:41999 every 8 connection(s)
I20260927 23:07:50.077566 20395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:07:50.083120 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 20395
Sep 27 23:07:50 dist-test-slave-lmfp krb5kdc[19492](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:07:50.098927 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.080030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:41621 (local address 127.3.240.254:45111)
0927 23:07:50.080243 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.080248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.081341 (+ 1093us) server_negotiation.cc:407] Connection header received
0927 23:07:50.083259 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.083265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.083360 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.083482 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:50.085508 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.086177 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.088126 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.088514 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.088600 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.092484 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:50.092488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:50.092491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:50.092525 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:50.095733 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.096494 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.096517 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.096523 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.096648 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.097060 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.097065 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.097069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.097360 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:50.097518 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.098299 (+ 781us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.098578 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260927 23:07:50.100008 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.083978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44448 (local address 127.3.240.254:45111)
0927 23:07:50.084779 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.084785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.084799 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:07:50.084849 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.084852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.084902 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.085013 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:50.086978 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.087703 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.089537 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.090057 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.090193 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.093515 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:50.093521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:50.093524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:50.093573 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:50.096673 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.097458 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.097466 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.097469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.097604 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.098145 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.098152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.098156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.098537 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:50.098818 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.099330 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.099562 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":698,"thread_start_us":589,"threads_started":1}
I20260927 23:07:50.100451 20525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.080450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:41621)
0927 23:07:50.081147 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:50.081195 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:50.082922 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:50.083718 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:50.083725 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:50.084584 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:07:50.085338 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.085364 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.086348 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.086355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.087988 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.088003 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.088732 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.088740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.089040 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.089839 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:07:50.089860 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:07:50.092282 (+ 2422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:07:50.095863 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:50.095871 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:50.095887 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:50.096360 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:50.096753 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:07:50.096758 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:07:50.096761 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:07:50.096941 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:07:50.097491 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:07:50.097499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:07:50.097823 (+ 324us) client_negotiation.cc:769] Sending connection context
0927 23:07:50.098150 (+ 327us) client_negotiation.cc:240] Negotiation successful
0927 23:07:50.098648 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":269,"threads_started":1}
I20260927 23:07:50.101943 20523 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45111
I20260927 23:07:50.102249 20523 heartbeater.cc:464] Registering TS with master...
I20260927 23:07:50.102921 20523 heartbeater.cc:511] Master 127.3.240.254:45111 requested a full tablet report, sending...
I20260927 23:07:50.104277 20075 ts_manager.cc:194] Registered new tserver with Master: 057416415fab4337a9a71718ecad7fac (127.3.240.195:41999)
I20260927 23:07:50.105026 20075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:41621
I20260927 23:07:50.114696 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:07:50.120833 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.115583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.3.240.254:45111)
0927 23:07:50.115739 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.115743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.116277 (+ 534us) server_negotiation.cc:407] Connection header received
0927 23:07:50.116451 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.116454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.116566 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.116671 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:07:50.117061 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.117668 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.119238 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.119622 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.119708 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.120202 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:07:50.120369 (+ 167us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0927 23:07:50.120658 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44454: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 23:07:50.120978 20527 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44454: token expired
W20260927 23:07:50.121526 19999 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.240.254:45111)
I20260927 23:07:50.133891 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.121993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44468 (local address 127.3.240.254:45111)
0927 23:07:50.122152 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.122159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.122176 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:07:50.122339 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.122344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.122400 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.122498 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:50.123887 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.124580 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.125899 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.126280 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.126413 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.128867 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:50.128873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:50.128877 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:50.128918 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:50.131199 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.131959 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.131967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.131971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.132111 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.132519 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.132524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.132528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.132803 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:50.133063 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.133364 (+ 301us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.133579 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260927 23:07:50.136950 20075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:07:50.143994 20075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:07:50.168851 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.154879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:35303 (local address 127.0.0.1:33802)
0927 23:07:50.160576 (+ 5697us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:50.160595 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:50.160695 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:50.162251 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:50.162256 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:50.162286 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:50.162773 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.162785 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.164610 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.164614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.166327 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.166342 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.167848 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.167854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.168063 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.168124 (+ 61us) client_negotiation.cc:769] Sending connection context
0927 23:07:50.168247 (+ 123us) client_negotiation.cc:240] Negotiation successful
0927 23:07:50.168421 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5576,"mutex_wait_us":9,"thread_start_us":4713,"threads_started":1}
I20260927 23:07:50.169039 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.156145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36796 (local address 127.3.240.193:45985)
0927 23:07:50.156840 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.156862 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.161223 (+ 4361us) server_negotiation.cc:407] Connection header received
0927 23:07:50.161535 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.161543 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.161768 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.161972 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:50.162562 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.163899 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.166541 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.167947 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.168072 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.168240 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.168557 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.168712 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":307,"threads_started":1}
I20260927 23:07:50.169509 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.155758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.3.240.195:41999)
0927 23:07:50.160864 (+ 5106us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.160873 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.160907 (+ 34us) server_negotiation.cc:407] Connection header received
0927 23:07:50.161004 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.161012 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.161252 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.162330 (+ 1078us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:50.162846 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.164433 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.167044 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.168376 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.168519 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.168723 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.168957 (+ 234us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.169205 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4922,"thread_start_us":4784,"threads_started":1}
I20260927 23:07:50.169561 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.155410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.3.240.194:35303)
0927 23:07:50.161515 (+ 6105us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.161522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.161550 (+ 28us) server_negotiation.cc:407] Connection header received
0927 23:07:50.161657 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.161663 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.161851 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.162031 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:50.162953 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.164434 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.166527 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.167647 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.167795 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.168021 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.169087 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.169249 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5937,"thread_start_us":5822,"threads_started":1}
I20260927 23:07:50.169646 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.154825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:41999 (local address 127.0.0.1:48924)
0927 23:07:50.159958 (+ 5133us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:50.159998 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:50.160143 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:50.161458 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:50.161466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:50.161498 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:50.161839 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.161857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.164560 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.164570 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.166838 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.166861 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.168595 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.168602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.168772 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.168812 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 23:07:50.168919 (+ 107us) client_negotiation.cc:240] Negotiation successful
0927 23:07:50.169262 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4913,"mutex_wait_us":43,"thread_start_us":5491,"threads_started":1}
I20260927 23:07:50.170290 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.155193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45985 (local address 127.0.0.1:36796)
0927 23:07:50.161131 (+ 5938us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:50.161145 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:50.161311 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:50.161955 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:50.161959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:50.161980 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:50.162360 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.162373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.164036 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.164039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.166361 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.166380 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.168161 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.168166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.168350 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.168379 (+ 29us) client_negotiation.cc:769] Sending connection context
0927 23:07:50.169530 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0927 23:07:50.169948 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5840,"mutex_wait_us":69,"thread_start_us":5720,"threads_started":1}
I20260927 23:07:50.173096 20327 tablet_service.cc:1467] Processing CreateTablet for tablet ef1fcb3fcc9448db81cc44b38b9c3c83 (DEFAULT_TABLE table=test-table [id=94d4ef434d1a478aa2ba35e37e59863d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:50.173224 20458 tablet_service.cc:1467] Processing CreateTablet for tablet ef1fcb3fcc9448db81cc44b38b9c3c83 (DEFAULT_TABLE table=test-table [id=94d4ef434d1a478aa2ba35e37e59863d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:50.173226 20195 tablet_service.cc:1467] Processing CreateTablet for tablet ef1fcb3fcc9448db81cc44b38b9c3c83 (DEFAULT_TABLE table=test-table [id=94d4ef434d1a478aa2ba35e37e59863d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:07:50.174312 20327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1fcb3fcc9448db81cc44b38b9c3c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:50.174445 20458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1fcb3fcc9448db81cc44b38b9c3c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:50.174451 20195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef1fcb3fcc9448db81cc44b38b9c3c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:07:50.185544 20539 tablet_bootstrap.cc:492] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Bootstrap starting.
I20260927 23:07:50.188474 20539 tablet_bootstrap.cc:654] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:50.188622 20540 tablet_bootstrap.cc:492] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Bootstrap starting.
I20260927 23:07:50.189846 20539 log.cc:824] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:50.190255 20541 tablet_bootstrap.cc:492] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Bootstrap starting.
I20260927 23:07:50.191967 20540 tablet_bootstrap.cc:654] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:50.193122 20540 log.cc:824] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:50.193459 20541 tablet_bootstrap.cc:654] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Neither blocks nor log segments found. Creating new log.
I20260927 23:07:50.193856 20539 tablet_bootstrap.cc:492] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: No bootstrap required, opened a new log
I20260927 23:07:50.194176 20539 ts_tablet_manager.cc:1397] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260927 23:07:50.194587 20541 log.cc:824] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Log is configured to *not* fsync() on all Append() calls
I20260927 23:07:50.201658 20540 tablet_bootstrap.cc:492] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: No bootstrap required, opened a new log
I20260927 23:07:50.201709 20541 tablet_bootstrap.cc:492] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: No bootstrap required, opened a new log
I20260927 23:07:50.201923 20540 ts_tablet_manager.cc:1397] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260927 23:07:50.201932 20541 ts_tablet_manager.cc:1397] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260927 23:07:50.206043 20539 raft_consensus.cc:348] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.206430 20539 raft_consensus.cc:374] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:50.206516 20539 raft_consensus.cc:729] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d450d031d53494c9c97e7948c85f3a1, State: Initialized, Role: FOLLOWER
I20260927 23:07:50.207104 20539 consensus_queue.cc:260] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.208197 20392 heartbeater.cc:503] Master 127.3.240.254:45111 was elected leader, sending a full tablet report...
I20260927 23:07:50.209692 20540 raft_consensus.cc:348] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.210062 20540 raft_consensus.cc:374] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:50.210069 20539 ts_tablet_manager.cc:1428] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260927 23:07:50.210170 20540 raft_consensus.cc:729] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e48f1927e3eb4f958f61e22a21a43086, State: Initialized, Role: FOLLOWER
I20260927 23:07:50.210691 20540 consensus_queue.cc:260] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.211782 20260 heartbeater.cc:503] Master 127.3.240.254:45111 was elected leader, sending a full tablet report...
I20260927 23:07:50.211918 20541 raft_consensus.cc:348] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.212378 20541 raft_consensus.cc:374] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:07:50.212404 20540 ts_tablet_manager.cc:1428] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260927 23:07:50.212471 20541 raft_consensus.cc:729] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 057416415fab4337a9a71718ecad7fac, State: Initialized, Role: FOLLOWER
I20260927 23:07:50.212980 20541 consensus_queue.cc:260] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.214315 20523 heartbeater.cc:503] Master 127.3.240.254:45111 was elected leader, sending a full tablet report...
I20260927 23:07:50.214982 20541 ts_tablet_manager.cc:1428] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
W20260927 23:07:50.227573 20261 tablet.cc:2369] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:07:50.329202 20524 tablet.cc:2369] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:07:50.384166 20545 raft_consensus.cc:482] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:07:50.384438 20545 raft_consensus.cc:504] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.385975 20545 leader_election.cc:288] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 057416415fab4337a9a71718ecad7fac (127.3.240.195:41999), e48f1927e3eb4f958f61e22a21a43086 (127.3.240.193:45985)
I20260927 23:07:50.393600 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.386436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:39289 (local address 127.3.240.193:45985)
0927 23:07:50.386617 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.386623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.387338 (+ 715us) server_negotiation.cc:407] Connection header received
0927 23:07:50.387562 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.387566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.387656 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.387770 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:50.388262 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.389269 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.391700 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.392773 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.392926 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.392996 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.393264 (+ 268us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.393349 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 23:07:50.393729 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.386297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45985 (local address 127.3.240.194:39289)
0927 23:07:50.387187 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:50.387212 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:50.387395 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:50.387787 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:50.387792 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:50.387817 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:50.388132 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.388141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.389409 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.389413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.391569 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.391583 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.392948 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.392951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.393109 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.393149 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 23:07:50.393240 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 23:07:50.393362 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":502,"threads_started":1}
I20260927 23:07:50.393782 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.386443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:46949 (local address 127.3.240.195:41999)
0927 23:07:50.386617 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.386623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.387649 (+ 1026us) server_negotiation.cc:407] Connection header received
0927 23:07:50.387839 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.387844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.387916 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.387976 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:07:50.388779 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.389653 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.391880 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.392982 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.393105 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.393148 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.393453 (+ 305us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.393564 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 23:07:50.394264 20549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.386352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:41999 (local address 127.3.240.194:46949)
0927 23:07:50.387547 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0927 23:07:50.387563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:07:50.387675 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:07:50.388173 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:07:50.388177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:07:50.388201 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:07:50.388639 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.388649 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.389775 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.389780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.391762 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:07:50.391771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.393155 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.393159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:07:50.393320 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.393344 (+ 24us) client_negotiation.cc:769] Sending connection context
0927 23:07:50.393413 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 23:07:50.393523 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1055,"thread_start_us":502,"threads_started":1}
I20260927 23:07:50.394793 20215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1fcb3fcc9448db81cc44b38b9c3c83" candidate_uuid: "0d450d031d53494c9c97e7948c85f3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48f1927e3eb4f958f61e22a21a43086" is_pre_election: true
I20260927 23:07:50.395381 20215 raft_consensus.cc:2454] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d450d031d53494c9c97e7948c85f3a1 in term 0.
I20260927 23:07:50.395268 20478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1fcb3fcc9448db81cc44b38b9c3c83" candidate_uuid: "0d450d031d53494c9c97e7948c85f3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057416415fab4337a9a71718ecad7fac" is_pre_election: true
I20260927 23:07:50.395766 20478 raft_consensus.cc:2454] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d450d031d53494c9c97e7948c85f3a1 in term 0.
I20260927 23:07:50.396055 20283 leader_election.cc:302] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d450d031d53494c9c97e7948c85f3a1, e48f1927e3eb4f958f61e22a21a43086; no voters:
I20260927 23:07:50.396473 20545 raft_consensus.cc:2781] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:07:50.396628 20545 raft_consensus.cc:482] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:07:50.396745 20545 raft_consensus.cc:3037] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:50.399102 20545 raft_consensus.cc:504] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.399721 20545 leader_election.cc:288] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [CANDIDATE]: Term 1 election: Requested vote from peers 057416415fab4337a9a71718ecad7fac (127.3.240.195:41999), e48f1927e3eb4f958f61e22a21a43086 (127.3.240.193:45985)
I20260927 23:07:50.400143 20478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1fcb3fcc9448db81cc44b38b9c3c83" candidate_uuid: "0d450d031d53494c9c97e7948c85f3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "057416415fab4337a9a71718ecad7fac"
I20260927 23:07:50.400341 20478 raft_consensus.cc:3037] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:50.402442 20478 raft_consensus.cc:2454] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d450d031d53494c9c97e7948c85f3a1 in term 1.
I20260927 23:07:50.402876 20281 leader_election.cc:302] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 057416415fab4337a9a71718ecad7fac, 0d450d031d53494c9c97e7948c85f3a1; no voters:
I20260927 23:07:50.402863 20215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef1fcb3fcc9448db81cc44b38b9c3c83" candidate_uuid: "0d450d031d53494c9c97e7948c85f3a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e48f1927e3eb4f958f61e22a21a43086"
I20260927 23:07:50.403081 20215 raft_consensus.cc:3037] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:07:50.403213 20545 raft_consensus.cc:2781] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:07:50.403623 20545 raft_consensus.cc:686] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 LEADER]: Becoming Leader. State: Replica: 0d450d031d53494c9c97e7948c85f3a1, State: Running, Role: LEADER
I20260927 23:07:50.404137 20545 consensus_queue.cc:237] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } }
I20260927 23:07:50.405575 20215 raft_consensus.cc:2454] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d450d031d53494c9c97e7948c85f3a1 in term 1.
I20260927 23:07:50.408241 20075 catalog_manager.cc:5495] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d450d031d53494c9c97e7948c85f3a1 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "0d450d031d53494c9c97e7948c85f3a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d450d031d53494c9c97e7948c85f3a1" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
W20260927 23:07:50.410099 20393 tablet.cc:2369] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:07:50.457423 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.452013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.3.240.194:35303)
0927 23:07:50.452214 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.452218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.452236 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:07:50.452457 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.452461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.452553 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.452706 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:07:50.453063 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.453818 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.455282 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.455723 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.455828 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.456268 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:07:50.456766 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:07:50.456891 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.456980 (+ 89us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.457045 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 23:07:50.465816 20478 raft_consensus.cc:1261] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 1 FOLLOWER]: Refusing update from remote peer 0d450d031d53494c9c97e7948c85f3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:07:50.465814 20215 raft_consensus.cc:1261] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 1 FOLLOWER]: Refusing update from remote peer 0d450d031d53494c9c97e7948c85f3a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:07:50.466724 20550 consensus_queue.cc:1035] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e48f1927e3eb4f958f61e22a21a43086" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:07:50.467118 20545 consensus_queue.cc:1035] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "057416415fab4337a9a71718ecad7fac" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:07:50.478763 20554 mvcc.cc:205] Tried to move back new op lower bound from 7334094727018811392 to 7334094726779686912. Current Snapshot: MvccSnapshot[applied={T|T < 7334094727018811392}]
I20260927 23:07:50.481046 20553 mvcc.cc:205] Tried to move back new op lower bound from 7334094727018811392 to 7334094726779686912. Current Snapshot: MvccSnapshot[applied={T|T < 7334094727018811392 or (T in {7334094727018811392})}]
I20260927 23:07:50.481887 20555 mvcc.cc:205] Tried to move back new op lower bound from 7334094727018811392 to 7334094726779686912. Current Snapshot: MvccSnapshot[applied={T|T < 7334094727018811392}]
I20260927 23:07:50.492719 20073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:07:50.493085 20073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:07:50.497087 20073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Sending DeleteTablet for 3 replicas of tablet ef1fcb3fcc9448db81cc44b38b9c3c83
I20260927 23:07:50.498178 20327 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1fcb3fcc9448db81cc44b38b9c3c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:50 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:33802
I20260927 23:07:50.498450 20195 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1fcb3fcc9448db81cc44b38b9c3c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:50 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36796
I20260927 23:07:50.498478 20458 tablet_service.cc:1514] Processing DeleteTablet for tablet ef1fcb3fcc9448db81cc44b38b9c3c83 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:07:50 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:48924
I20260927 23:07:50.499715 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 20112
I20260927 23:07:50.507509 20563 tablet_replica.cc:330] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: stopping tablet replica
I20260927 23:07:50.508066 20563 raft_consensus.cc:2229] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:07:50.508747 20563 raft_consensus.cc:2258] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:50.508790 20565 tablet_replica.cc:330] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: stopping tablet replica
I20260927 23:07:50.509202 20565 raft_consensus.cc:2229] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:07:50.509495 20564 tablet_replica.cc:330] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: stopping tablet replica
I20260927 23:07:50.509723 20565 raft_consensus.cc:2258] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:50.509881 20564 raft_consensus.cc:2229] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:07:50.510373 20564 raft_consensus.cc:2258] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:07:50.512643 20564 ts_tablet_manager.cc:1905] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:50.512763 20563 ts_tablet_manager.cc:1905] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:50.520251 20564 ts_tablet_manager.cc:1918] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:50.520421 20564 log.cc:1196] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/ef1fcb3fcc9448db81cc44b38b9c3c83
I20260927 23:07:50.520987 20564 ts_tablet_manager.cc:1939] T ef1fcb3fcc9448db81cc44b38b9c3c83 P e48f1927e3eb4f958f61e22a21a43086: Deleting consensus metadata
I20260927 23:07:50.521086 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.501900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.3.240.193:45985)
0927 23:07:50.502114 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.502122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.502144 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:07:50.502212 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.502219 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.502314 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.502442 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:50.504962 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.506518 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.509694 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.510274 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.510449 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.514280 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:50.514293 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:50.514319 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:50.514350 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:50.517631 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.518574 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.518580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.518582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.518780 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.519266 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.519271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.519275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.519657 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:50.519972 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.520289 (+ 317us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.520542 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 23:07:50.521831 20255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:50.522720 20565 ts_tablet_manager.cc:1905] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:07:50.527875 20563 ts_tablet_manager.cc:1918] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:50.528062 20563 log.cc:1196] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/ef1fcb3fcc9448db81cc44b38b9c3c83
I20260927 23:07:50.528612 20563 ts_tablet_manager.cc:1939] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 0d450d031d53494c9c97e7948c85f3a1: Deleting consensus metadata
I20260927 23:07:50.530951 20061 catalog_manager.cc:4841] TS 0d450d031d53494c9c97e7948c85f3a1 (127.3.240.194:35303): tablet ef1fcb3fcc9448db81cc44b38b9c3c83 (table test-table [id=94d4ef434d1a478aa2ba35e37e59863d]) successfully deleted
I20260927 23:07:50.532158 20565 ts_tablet_manager.cc:1918] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:07:50.532337 20565 log.cc:1196] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/ef1fcb3fcc9448db81cc44b38b9c3c83
I20260927 23:07:50.532876 20565 ts_tablet_manager.cc:1939] T ef1fcb3fcc9448db81cc44b38b9c3c83 P 057416415fab4337a9a71718ecad7fac: Deleting consensus metadata
I20260927 23:07:50.534701 20060 catalog_manager.cc:4841] TS 057416415fab4337a9a71718ecad7fac (127.3.240.195:41999): tablet ef1fcb3fcc9448db81cc44b38b9c3c83 (table test-table [id=94d4ef434d1a478aa2ba35e37e59863d]) successfully deleted
I20260927 23:07:50.640879 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 20112
I20260927 23:07:50.640944 20062 catalog_manager.cc:4841] TS e48f1927e3eb4f958f61e22a21a43086 (127.3.240.193:45985): tablet ef1fcb3fcc9448db81cc44b38b9c3c83 (table test-table [id=94d4ef434d1a478aa2ba35e37e59863d]) successfully deleted
I20260927 23:07:50.658948 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 20264
I20260927 23:07:50.674216 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.659748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.3.240.194:35303)
0927 23:07:50.659940 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.659946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.660105 (+ 159us) server_negotiation.cc:407] Connection header received
0927 23:07:50.660321 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.660325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.660393 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.660520 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:50.661944 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.662782 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.664263 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.664742 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.664885 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.667602 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:50.667613 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:50.667626 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:50.667677 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:50.671275 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.671933 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.671945 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.671953 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.672171 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.672616 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.672621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.672625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.672961 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:50.673299 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.673585 (+ 286us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.673786 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:07:50.674908 20387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:50.789062 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 20264
I20260927 23:07:50.808422 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 20395
I20260927 23:07:50.821326 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:07:50.809178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.3.240.195:41999)
0927 23:07:50.809366 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 23:07:50.809370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:07:50.809387 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:07:50.809545 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:07:50.809549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:07:50.809610 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:07:50.809691 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:07:50.811182 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.811971 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.813215 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:07:50.813598 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:07:50.813697 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:07:50.816215 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:07:50.816228 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:07:50.816243 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:07:50.816282 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:07:50.818763 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.819436 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.819444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.819446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.819582 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:07:50.819916 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:07:50.819919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:07:50.819921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:07:50.820237 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:07:50.820447 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0927 23:07:50.820794 (+ 347us) server_negotiation.cc:299] Negotiation successful
0927 23:07:50.821030 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:07:50.822060 20518 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:50.911095 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 20395
I20260927 23:07:50.928901 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 20042
I20260927 23:07:50.932734 20105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:07:51.063741 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 20042
I20260927 23:08:11.086027 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45111
--webserver_interface=127.3.240.254
--webserver_port=35639
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:11.399225 20570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:11.399474 20570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:11.399554 20570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:11.406147 20570 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260927 23:08:11.406224 20570 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260927 23:08:11.406256 20570 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260927 23:08:11.406302 20570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:11.406348 20570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:11.406401 20570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:11.406441 20570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:11.406483 20570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:11.414870 20570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=35639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:11.415896 20570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:11.416985 20570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:11.428486 20575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:11.428445 20578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:11.428529 20576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:11.432276 20570 server_base.cc:1034] running on GCE node
I20260927 23:08:11.432935 20570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:11.434022 20570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:11.435212 20570 hybrid_clock.cc:648] HybridClock initialized: now 1790550491435165 us; error 35 us; skew 500 ppm
I20260927 23:08:11.435508 20570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:11 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550491, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:11.438215 20570 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:08:11.439625 20570 webserver.cc:460] Webserver started at http://127.3.240.254:35639/ using document root <none> and password file <none>
I20260927 23:08:11.440256 20570 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:11.440382 20570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:11.445148 20570 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260927 23:08:11.448027 20585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:11.449141 20570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:08:11.449314 20570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "14c7b77c729849479fcbc2041e851bbf"
format_stamp: "Formatted at 2026-09-27 23:07:25 on dist-test-slave-lmfp"
server_key: "ff853b49d2514f15f24971097c29ee6c"
server_key_iv: "920ddc82cd9d8b458cc8764f3cecf915"
server_key_version: "encryptionkey@0"
I20260927 23:08:11.450294 20570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:11.468307 20570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:11.472539 20570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:11.472862 20570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:11.490936 20570 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:45111
I20260927 23:08:11.490967 20636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:45111 every 8 connection(s)
I20260927 23:08:11.492319 20570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:11.496743 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 20570
I20260927 23:08:11.497493 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:45985
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:45111
--webserver_port=37953
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:08:11.498924 20637 sys_catalog.cc:263] Verifying existing consensus state
I20260927 23:08:11.502048 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Bootstrap starting.
I20260927 23:08:11.522075 20637 log.cc:824] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:11.536957 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260927 23:08:11.537456 20637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Bootstrap complete.
I20260927 23:08:11.546178 20637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:08:11.546947 20637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c7b77c729849479fcbc2041e851bbf, State: Initialized, Role: FOLLOWER
I20260927 23:08:11.547539 20637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:08:11.547784 20637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:11.547914 20637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:11.548096 20637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 2 FOLLOWER]: Advancing to term 3
I20260927 23:08:11.551712 20637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:08:11.552143 20637 leader_election.cc:302] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14c7b77c729849479fcbc2041e851bbf; no voters:
I20260927 23:08:11.552757 20637 leader_election.cc:288] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260927 23:08:11.553028 20641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 3 FOLLOWER]: Leader election won for term 3
I20260927 23:08:11.554108 20641 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [term 3 LEADER]: Becoming Leader. State: Replica: 14c7b77c729849479fcbc2041e851bbf, State: Running, Role: LEADER
I20260927 23:08:11.554782 20637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:11.554702 20641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } }
I20260927 23:08:11.557006 20642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "14c7b77c729849479fcbc2041e851bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } } }
I20260927 23:08:11.557331 20642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:11.558681 20643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14c7b77c729849479fcbc2041e851bbf. Latest consensus state: current_term: 3 leader_uuid: "14c7b77c729849479fcbc2041e851bbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c7b77c729849479fcbc2041e851bbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45111 } } }
I20260927 23:08:11.558984 20643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:11.560531 20646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:11.570113 20646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:11.571976 20646 catalog_manager.cc:1259] Loaded cluster ID: f9df7fb6922640eebaff7a77012d082c
I20260927 23:08:11.572425 20646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:11.577605 20646 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:11.579545 20646 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Loaded TSK: 0
I20260927 23:08:11.579780 20646 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14c7b77c729849479fcbc2041e851bbf: Loaded TSK: 1
F20260927 23:08:11.581193 20646 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f431b9f29ad google::LogMessage::Fail() at ??:0
I20260927 23:08:11.585444 20658 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f431b9f588a google::LogMessage::SendToLog() at ??:0
@ 0x7f431b9f24de google::LogMessage::Flush() at ??:0
@ 0x7f431b9f3a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f431f7567d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f431f74573e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f43314211c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4331420783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f43314ac2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f43314ac0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f43314f65e9 std::function<>::operator()() at ??:0
@ 0x7f4331424cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4331422c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f43314aa610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f43314aa439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f431e63e345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f431e64191c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f431e641799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f431e61ceaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f43218f66db start_thread at ??:0
@ 0x7f431a88c61f clone at ??:0
W20260927 23:08:11.869730 20639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:11.869980 20639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:11.870069 20639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:11.877099 20639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:11.877205 20639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:11.877321 20639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:08:11.886729 20639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:45985
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=37953
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:11.888022 20639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:11.889284 20639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:11.902177 20684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:11.902226 20687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:11.902268 20685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:11.903339 20639 server_base.cc:1034] running on GCE node
I20260927 23:08:11.903909 20639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:11.904690 20639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:11.905901 20639 hybrid_clock.cc:648] HybridClock initialized: now 1790550491905862 us; error 35 us; skew 500 ppm
I20260927 23:08:11.906210 20639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:11 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550491, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:11.909080 20639 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:08:11.910452 20639 webserver.cc:460] Webserver started at http://127.3.240.193:37953/ using document root <none> and password file <none>
I20260927 23:08:11.911163 20639 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:11.911305 20639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:11.916097 20639 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:08:11.918519 20694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:11.919783 20639 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:08:11.919960 20639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "e48f1927e3eb4f958f61e22a21a43086"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "3aa6f4a404d99682b5ec63e5e3a3d593"
server_key_iv: "132e438eccf1cdf302131c97f6c52312"
server_key_version: "encryptionkey@0"
I20260927 23:08:11.920869 20639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:11.934386 20639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:11.937917 20639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:11.938179 20639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:11.939191 20639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:11.940413 20639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:11.940531 20639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:11.940661 20639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:11.940747 20639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:11.974843 20639 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:45985
I20260927 23:08:11.974891 20806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:45985 every 8 connection(s)
I20260927 23:08:11.976380 20639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:08:11.979802 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 20639
I20260927 23:08:11.980418 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:35303
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:45111
--webserver_port=35123
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:08:11.981727 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:11.979637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:55189)
0927 23:08:11.980493 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:11.980811 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":356,"threads_started":1}
W20260927 23:08:11.982432 20807 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:45111 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:45111: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
I20260927 23:08:11.983392 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:11.983063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:41545)
0927 23:08:11.983210 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:11.983282 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260927 23:08:11.984331 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:11.983985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:46383)
0927 23:08:11.984143 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:11.984220 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 23:08:11.984591 20807 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260927 23:08:12.298053 20810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:12.298302 20810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:12.298398 20810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:12.305150 20810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:12.305254 20810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:12.305366 20810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:08:12.314272 20810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:35303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=35123
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:12.315442 20810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:12.316592 20810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:12.329447 20815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:12.329422 20816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:12.329537 20818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:12.330255 20810 server_base.cc:1034] running on GCE node
I20260927 23:08:12.330868 20810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:12.331772 20810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:12.332988 20810 hybrid_clock.cc:648] HybridClock initialized: now 1790550492332936 us; error 41 us; skew 500 ppm
I20260927 23:08:12.333312 20810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:12 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:12.336356 20810 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:08:12.337779 20810 webserver.cc:460] Webserver started at http://127.3.240.194:35123/ using document root <none> and password file <none>
I20260927 23:08:12.338424 20810 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:12.338565 20810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:12.343631 20810 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:08:12.346175 20825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:12.347401 20810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260927 23:08:12.347579 20810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "0d450d031d53494c9c97e7948c85f3a1"
format_stamp: "Formatted at 2026-09-27 23:07:26 on dist-test-slave-lmfp"
server_key: "8160e73066aab1d8ccd3d7479903c02e"
server_key_iv: "2cf1e3f1bff5acdd8f9383dd28d442ff"
server_key_version: "encryptionkey@0"
I20260927 23:08:12.348532 20810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:12.393134 20810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:12.396781 20810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:12.397030 20810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:12.397938 20810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:12.399181 20810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:12.399292 20810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:12.399418 20810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:12.399503 20810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:12.432628 20810 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:35303
I20260927 23:08:12.432682 20937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:35303 every 8 connection(s)
I20260927 23:08:12.434077 20810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:08:12.438889 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:12.437007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:54701)
0927 23:08:12.437820 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:12.438090 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":347,"threads_started":1}
W20260927 23:08:12.439667 20938 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:45111 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:45111: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
I20260927 23:08:12.440431 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:12.440138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:57735)
0927 23:08:12.440272 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:12.440336 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 23:08:12.441193 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:12.440910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:56215)
0927 23:08:12.441018 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:12.441083 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 23:08:12.441452 20938 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 23:08:12.442763 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 20810
I20260927 23:08:12.443408 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:41999
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:45111
--webserver_port=36641
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:36341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:12.749245 20941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:12.749507 20941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:12.749598 20941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:12.756541 20941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:12.756655 20941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:12.756772 20941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:08:12.765692 20941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:36341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:41999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=36641
--tserver_master_addrs=127.3.240.254:45111
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:12.766791 20941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:12.768085 20941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:12.780586 20946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:12.780630 20949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:12.780544 20947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:12.781493 20941 server_base.cc:1034] running on GCE node
I20260927 23:08:12.782043 20941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:12.782831 20941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:12.784061 20941 hybrid_clock.cc:648] HybridClock initialized: now 1790550492784014 us; error 59 us; skew 500 ppm
I20260927 23:08:12.784358 20941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:12 dist-test-slave-lmfp krb5kdc[19492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550492, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:12.787200 20941 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:08:12.788520 20941 webserver.cc:460] Webserver started at http://127.3.240.195:36641/ using document root <none> and password file <none>
I20260927 23:08:12.789139 20941 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:12.789284 20941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:12.794370 20941 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:08:12.797034 20956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:12.798277 20941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 23:08:12.798448 20941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "057416415fab4337a9a71718ecad7fac"
format_stamp: "Formatted at 2026-09-27 23:07:27 on dist-test-slave-lmfp"
server_key: "5db8e2c3cbb9f9ee5c0e0f93e8c1dfdb"
server_key_iv: "d97335439690bf939030ca5f2aa951ed"
server_key_version: "encryptionkey@0"
I20260927 23:08:12.799399 20941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:12.813215 20941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:12.816876 20941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:12.817131 20941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:12.818037 20941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:12.819298 20941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:12.819423 20941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:12.819550 20941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:12.819638 20941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:12.851517 20941 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:41999
I20260927 23:08:12.851570 21068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:41999 every 8 connection(s)
I20260927 23:08:12.853040 20941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:08:12.853502 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 20941
I20260927 23:08:12.857626 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:12.855992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:35393)
0927 23:08:12.856659 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:12.856855 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":261,"threads_started":1}
W20260927 23:08:12.858242 21069 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:45111 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:45111: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
I20260927 23:08:12.858973 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:12.858676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:44947)
0927 23:08:12.858811 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:12.858881 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260927 23:08:12.859827 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:12.859518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:50125)
0927 23:08:12.859646 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:12.859728 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 23:08:12.860050 21069 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260927 23:08:12.986146 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:12.985236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:38395)
0927 23:08:12.985768 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:12.985867 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":288,"threads_started":1}
I20260927 23:08:13.442759 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:13.441938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:36253)
0927 23:08:13.442407 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:13.442496 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":224,"threads_started":1}
I20260927 23:08:13.861263 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:13.860533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:59585)
0927 23:08:13.860942 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:13.861040 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":204,"threads_started":1}
I20260927 23:08:13.987582 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:13.986904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:52711)
0927 23:08:13.987360 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:13.987433 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":247,"threads_started":1}
I20260927 23:08:14.444396 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:14.443631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:53593)
0927 23:08:14.444132 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:14.444219 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":276,"threads_started":1}
I20260927 23:08:14.862627 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:14.861998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:58211)
0927 23:08:14.862410 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:14.862481 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":213,"threads_started":1}
I20260927 23:08:14.988981 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:14.988364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:53333)
0927 23:08:14.988738 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:14.988807 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":186,"threads_started":1}
I20260927 23:08:15.446044 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:15.445312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:40149)
0927 23:08:15.445749 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:15.445842 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":218,"threads_started":1}
I20260927 23:08:15.864089 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:15.863436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:46035)
0927 23:08:15.863860 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:15.863933 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260927 23:08:15.990783 21082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:15.989936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:45133)
0927 23:08:15.990423 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:15.990534 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":249,"threads_started":1}
I20260927 23:08:16.447722 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:16.446983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:43119)
0927 23:08:16.447490 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:16.447568 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":249,"threads_started":1}
I20260927 23:08:16.865507 21084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:16.864853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:54757)
0927 23:08:16.865263 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:16.865338 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":201,"threads_started":1}
I20260927 23:08:16.992301 21085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:16.991599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:34611)
0927 23:08:16.992059 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:16.992146 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":260,"threads_started":1}
I20260927 23:08:17.449301 21086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:17.448590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:45727)
0927 23:08:17.449052 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:17.449126 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":258,"threads_started":1}
I20260927 23:08:17.867378 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:17.866448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:36167)
0927 23:08:17.866961 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:17.867183 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":276,"threads_started":1}
I20260927 23:08:17.993820 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:17.993117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:45943)
0927 23:08:17.993567 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:17.993652 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":228,"threads_started":1}
I20260927 23:08:18.450879 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:18.450181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:56089)
0927 23:08:18.450630 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:18.450722 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":240,"threads_started":1}
I20260927 23:08:18.868964 21090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:18.868294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:47575)
0927 23:08:18.868727 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:18.868814 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":230,"threads_started":1}
I20260927 23:08:18.995705 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:18.994792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:48157)
0927 23:08:18.995404 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:18.995498 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":270,"threads_started":1}
I20260927 23:08:19.452844 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:19.451897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:35385)
0927 23:08:19.452491 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:19.452585 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":250,"threads_started":1}
I20260927 23:08:19.870708 21093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:19.869918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:52705)
0927 23:08:19.870431 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:19.870504 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":238,"threads_started":1}
I20260927 23:08:19.997349 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:19.996577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:44803)
0927 23:08:19.997083 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:19.997154 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260927 23:08:20.454422 21095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:20.453641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:58317)
0927 23:08:20.454122 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:20.454225 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":255,"threads_started":1}
I20260927 23:08:20.872294 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:20.871548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:54435)
0927 23:08:20.872058 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:20.872137 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":246,"threads_started":1}
I20260927 23:08:20.998966 21097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:20.998180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:52363)
0927 23:08:20.998730 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:20.998805 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":299,"threads_started":1}
I20260927 23:08:21.455912 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:21.455260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:39957)
0927 23:08:21.455667 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:21.455740 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":202,"threads_started":1}
I20260927 23:08:21.873764 21099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:21.873051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:48265)
0927 23:08:21.873524 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:21.873598 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":201,"threads_started":1}
I20260927 23:08:22.000458 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:21.999830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:49987)
0927 23:08:22.000216 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:22.000286 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":209,"threads_started":1}
I20260927 23:08:22.457477 21101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:22.456759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:43593)
0927 23:08:22.457232 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:22.457316 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
I20260927 23:08:22.875402 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:22.874614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:51903)
0927 23:08:22.875156 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:22.875242 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":268,"threads_started":1}
I20260927 23:08:23.001873 21103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:23.001194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:56875)
0927 23:08:23.001624 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:23.001708 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260927 23:08:23.459268 21104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:23.458486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:55851)
0927 23:08:23.458946 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:23.459090 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260927 23:08:23.876857 21105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:23.876170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:40303)
0927 23:08:23.876562 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:23.876670 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":204,"threads_started":1}
I20260927 23:08:24.003728 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:24.002923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:53359)
0927 23:08:24.003473 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:24.003556 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":235,"threads_started":1}
I20260927 23:08:24.460728 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:24.460083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:37273)
0927 23:08:24.460482 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:24.460555 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":216,"threads_started":1}
I20260927 23:08:24.878335 21108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:24.877650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:39311)
0927 23:08:24.878118 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:24.878191 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":266,"threads_started":1}
I20260927 23:08:25.005293 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:25.004560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:57973)
0927 23:08:25.005011 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:25.005103 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":223,"threads_started":1}
I20260927 23:08:25.462273 21110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:25.461520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:57569)
0927 23:08:25.461998 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:25.462088 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":266,"threads_started":1}
I20260927 23:08:25.879918 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:25.879241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:34685)
0927 23:08:25.879665 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:25.879752 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":206,"threads_started":1}
I20260927 23:08:26.006886 21112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:26.006229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:45691)
0927 23:08:26.006628 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:26.006704 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":201,"threads_started":1}
I20260927 23:08:26.463740 21113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:26.463104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:53839)
0927 23:08:26.463480 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:26.463563 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":189,"threads_started":1}
I20260927 23:08:26.881335 21114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:26.880649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:33985)
0927 23:08:26.881076 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:26.881158 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":231,"threads_started":1}
I20260927 23:08:27.008558 21115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:27.007905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.193:51965)
0927 23:08:27.008303 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:27.008386 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":199,"threads_started":1}
I20260927 23:08:27.465504 21116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:27.464824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.194:52875)
0927 23:08:27.465230 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:27.465329 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":215,"threads_started":1}
I20260927 23:08:27.861029 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 20639
I20260927 23:08:27.883080 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:27.882317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45111 (local address 127.3.240.195:45791)
0927 23:08:27.882723 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:27.882805 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:45111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":198,"threads_started":1}
I20260927 23:08:27.884750 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:27.862111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.3.240.193:45985)
0927 23:08:27.863369 (+ 1258us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:27.863384 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:27.863420 (+ 36us) server_negotiation.cc:407] Connection header received
0927 23:08:27.864789 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:27.864797 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:27.865208 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:27.865478 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:27.866659 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:27.867893 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:27.869075 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:27.869709 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:27.869843 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:27.878207 (+ 8364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:27.878216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:27.878231 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:27.878270 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:27.881186 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:27.881846 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:27.881852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:27.881858 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:27.882016 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:27.882957 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:27.882962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:27.882965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:27.883614 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:27.883835 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:27.884166 (+ 331us) server_negotiation.cc:299] Negotiation successful
0927 23:08:27.884480 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":993,"threads_started":1}
I20260927 23:08:27.886065 20802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:27.966837 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 20639
I20260927 23:08:27.984100 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 20810
I20260927 23:08:28.000725 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:27.985020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.3.240.194:35303)
0927 23:08:27.985537 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:27.985550 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:27.985588 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:08:27.987230 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:27.987240 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:27.987707 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:27.987999 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:27.989290 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:27.990475 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:27.991759 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:27.992430 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:27.992581 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:27.994862 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:27.994869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:27.994883 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:27.994925 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:27.997974 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:27.998648 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:27.998654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:27.998660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:27.998827 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:27.999195 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:27.999198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:27.999199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:27.999663 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:27.999878 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:28.000145 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 23:08:28.000429 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260927 23:08:28.002086 20933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:28.084060 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 20810
I20260927 23:08:28.100960 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 20941
I20260927 23:08:28.118505 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:28.101989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36614 (local address 127.3.240.195:41999)
0927 23:08:28.102542 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:28.102556 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:28.102598 (+ 42us) server_negotiation.cc:407] Connection header received
0927 23:08:28.104337 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:28.104346 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:28.104763 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:28.105147 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:28.106563 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:28.107921 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:28.109141 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:28.109786 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:28.109968 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:28.112505 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:28.112513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:28.112527 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:28.112573 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:28.115247 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:28.115912 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:28.115920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:28.115929 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:28.116097 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:28.116488 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:28.116494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:28.116497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:28.116993 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:28.117297 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:28.117748 (+ 451us) server_negotiation.cc:299] Negotiation successful
0927 23:08:28.118160 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":254,"threads_started":1}
I20260927 23:08:28.119958 21064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:28.203159 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 20941
2026-09-27T23:08:28Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64147 ms)
[----------] 30 tests from SecurityITest (283741 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:28 dist-test-slave-lmfp krb5kdc[21126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:28 dist-test-slave-lmfp krb5kdc[21126](info): set up 2 sockets
Sep 27 23:08:28 dist-test-slave-lmfp krb5kdc[21126](info): commencing operation
krb5kdc: starting...
W20260927 23:08:29.287139 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:29 dist-test-slave-lmfp krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:08:29.524601 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40441
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:40441
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:29.850389 21139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:29.850662 21139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:29.850831 21139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:29.857821 21139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:29.857909 21139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:29.858021 21139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:29.858119 21139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:29.858194 21139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:29.867549 21139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:40441
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:29.868774 21139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:29.870075 21139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:29.881865 21144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:29.882000 21147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:29.881889 21145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:29.882696 21139 server_base.cc:1034] running on GCE node
I20260927 23:08:29.883478 21139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:29.884863 21139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:29.886101 21139 hybrid_clock.cc:648] HybridClock initialized: now 1790550509886059 us; error 51 us; skew 500 ppm
I20260927 23:08:29.886449 21139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:29 dist-test-slave-lmfp krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550509, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:29.889463 21139 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:08:29.890959 21139 webserver.cc:460] Webserver started at http://127.3.240.254:34941/ using document root <none> and password file <none>
I20260927 23:08:29.891691 21139 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:29.891834 21139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:29.892144 21139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:29.894985 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "c490819d6588469a9a93355cbb37988d"
format_stamp: "Formatted at 2026-09-27 23:08:29 on dist-test-slave-lmfp"
server_key: "5320c53f65a5c42bef93d62dcf2f42ca"
server_key_iv: "418112cdff600e037100c316810d7f72"
server_key_version: "encryptionkey@0"
I20260927 23:08:29.895815 21139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "c490819d6588469a9a93355cbb37988d"
format_stamp: "Formatted at 2026-09-27 23:08:29 on dist-test-slave-lmfp"
server_key: "5320c53f65a5c42bef93d62dcf2f42ca"
server_key_iv: "418112cdff600e037100c316810d7f72"
server_key_version: "encryptionkey@0"
I20260927 23:08:29.900795 21139 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:08:29.904212 21154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:29.905571 21139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:08:29.905750 21139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "c490819d6588469a9a93355cbb37988d"
format_stamp: "Formatted at 2026-09-27 23:08:29 on dist-test-slave-lmfp"
server_key: "5320c53f65a5c42bef93d62dcf2f42ca"
server_key_iv: "418112cdff600e037100c316810d7f72"
server_key_version: "encryptionkey@0"
I20260927 23:08:29.905941 21139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:29.922406 21139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:29.926288 21139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:29.926554 21139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:29.944145 21139 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:40441
I20260927 23:08:29.944192 21205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:40441 every 8 connection(s)
I20260927 23:08:29.945439 21139 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:29.946789 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 21139
I20260927 23:08:29.947045 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:29.947969 4035 external_mini_cluster.cc:1428] Setting key 790aef154f8fee01c5b9fc07e50568e0
I20260927 23:08:29.950064 21206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 23:08:29 dist-test-slave-lmfp krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:29.963274 21206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d: Bootstrap starting.
I20260927 23:08:29.967350 21206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:29.968515 21206 log.cc:824] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:29.971743 21206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d: No bootstrap required, opened a new log
I20260927 23:08:29.971972 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:29.949920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.3.240.254:40441)
0927 23:08:29.950716 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:29.950745 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:29.950786 (+ 41us) server_negotiation.cc:407] Connection header received
0927 23:08:29.952421 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:29.952441 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:29.952801 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:29.953139 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:29.954867 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:29.956202 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:29.958010 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:29.958614 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:29.958732 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:29.963694 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:29.963704 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:29.963723 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:29.963764 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:29.967702 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:29.968398 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:29.968405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:29.968412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:29.968618 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:29.969015 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:29.969023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:29.969030 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:29.969554 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:29.969821 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:29.970261 (+ 440us) server_negotiation.cc:299] Negotiation successful
0927 23:08:29.970675 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":409,"threads_started":1}
I20260927 23:08:29.979542 21206 raft_consensus.cc:348] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c490819d6588469a9a93355cbb37988d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40441 } }
I20260927 23:08:29.979847 21206 raft_consensus.cc:374] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:29.979918 21206 raft_consensus.cc:729] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c490819d6588469a9a93355cbb37988d, State: Initialized, Role: FOLLOWER
I20260927 23:08:29.980399 21206 consensus_queue.cc:260] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c490819d6588469a9a93355cbb37988d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40441 } }
I20260927 23:08:29.980590 21206 raft_consensus.cc:388] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:29.980680 21206 raft_consensus.cc:482] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:29.980811 21206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:29.983098 21206 raft_consensus.cc:504] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c490819d6588469a9a93355cbb37988d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40441 } }
I20260927 23:08:29.983505 21206 leader_election.cc:302] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c490819d6588469a9a93355cbb37988d; no voters:
I20260927 23:08:29.984045 21206 leader_election.cc:288] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:29.984210 21211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:29.984887 21211 raft_consensus.cc:686] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [term 1 LEADER]: Becoming Leader. State: Replica: c490819d6588469a9a93355cbb37988d, State: Running, Role: LEADER
I20260927 23:08:29.985512 21211 consensus_queue.cc:237] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c490819d6588469a9a93355cbb37988d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40441 } }
I20260927 23:08:29.986157 21206 sys_catalog.cc:564] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:29.988845 21213 sys_catalog.cc:455] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c490819d6588469a9a93355cbb37988d. Latest consensus state: current_term: 1 leader_uuid: "c490819d6588469a9a93355cbb37988d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c490819d6588469a9a93355cbb37988d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40441 } } }
I20260927 23:08:29.988893 21212 sys_catalog.cc:455] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c490819d6588469a9a93355cbb37988d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c490819d6588469a9a93355cbb37988d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40441 } } }
I20260927 23:08:29.989387 21213 sys_catalog.cc:458] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:29.989384 21212 sys_catalog.cc:458] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:29.991775 21218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:29.998032 21218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:30.007709 21218 catalog_manager.cc:1347] Generated new cluster ID: 611591db4c4a441e8a15b3ffb06ecdd4
I20260927 23:08:30.007938 21218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:30.021510 21218 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:30.022782 21218 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:30.032541 21218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d: Generated new TSK 0
I20260927 23:08:30.033614 21218 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:08:30.094009 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40441
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:30.413990 21232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:30.414244 21232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:30.414402 21232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:30.421403 21232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:30.421511 21232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:30.421701 21232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:08:30.431433 21232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:30.432651 21232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:30.433858 21232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:30.446580 21237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:30.446556 21238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:30.446498 21240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:30.447630 21232 server_base.cc:1034] running on GCE node
I20260927 23:08:30.448272 21232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:30.449208 21232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:30.450423 21232 hybrid_clock.cc:648] HybridClock initialized: now 1790550510450383 us; error 53 us; skew 500 ppm
I20260927 23:08:30.450728 21232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:30 dist-test-slave-lmfp krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550510, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:30.453819 21232 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:08:30.455153 21232 webserver.cc:460] Webserver started at http://127.3.240.193:42989/ using document root <none> and password file <none>
I20260927 23:08:30.455756 21232 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:30.455848 21232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:30.456087 21232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:30.458905 21232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "66cbbe99ab0f400b846075579d10fa1b"
format_stamp: "Formatted at 2026-09-27 23:08:30 on dist-test-slave-lmfp"
server_key: "251b5ab8f81c5bdd3fe405ed132511be"
server_key_iv: "4d1c6772727f0609ce2db7dab2b0d65b"
server_key_version: "encryptionkey@0"
I20260927 23:08:30.459673 21232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "66cbbe99ab0f400b846075579d10fa1b"
format_stamp: "Formatted at 2026-09-27 23:08:30 on dist-test-slave-lmfp"
server_key: "251b5ab8f81c5bdd3fe405ed132511be"
server_key_iv: "4d1c6772727f0609ce2db7dab2b0d65b"
server_key_version: "encryptionkey@0"
I20260927 23:08:30.464285 21232 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260927 23:08:30.467411 21247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:30.468590 21232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260927 23:08:30.468735 21232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "66cbbe99ab0f400b846075579d10fa1b"
format_stamp: "Formatted at 2026-09-27 23:08:30 on dist-test-slave-lmfp"
server_key: "251b5ab8f81c5bdd3fe405ed132511be"
server_key_iv: "4d1c6772727f0609ce2db7dab2b0d65b"
server_key_version: "encryptionkey@0"
I20260927 23:08:30.468920 21232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:30.481995 21232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:30.486029 21232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:30.486306 21232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:30.487427 21232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:30.488672 21232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:30.488740 21232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:30.488888 21232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:30.488960 21232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:30.521229 21232 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:37629
I20260927 23:08:30.521282 21359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:37629 every 8 connection(s)
I20260927 23:08:30.522451 21232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:08:30.527699 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 21232
I20260927 23:08:30.527897 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:08:30.528553 4035 external_mini_cluster.cc:1428] Setting key 0f317092d23671f715ce2fc7390f3b94
Sep 27 23:08:30 dist-test-slave-lmfp krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550510, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:30.542801 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:30.524892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:35517 (local address 127.3.240.254:40441)
0927 23:08:30.525321 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:30.525326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:30.526083 (+ 757us) server_negotiation.cc:407] Connection header received
0927 23:08:30.528022 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:30.528028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:30.528121 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:30.528231 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:30.529797 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:30.530602 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:30.532082 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:30.532503 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:30.532662 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:30.536368 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:30.536372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:30.536375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:30.536405 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:30.539653 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:30.540309 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:30.540317 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:30.540323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:30.540475 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:30.540862 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:30.540869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:30.540875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:30.541254 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:30.541426 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:30.542057 (+ 631us) server_negotiation.cc:299] Negotiation successful
0927 23:08:30.542302 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":232,"threads_started":1}
I20260927 23:08:30.543715 21363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:30.525277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40441 (local address 127.3.240.193:35517)
0927 23:08:30.525920 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:30.525961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:30.527731 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:30.528450 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:30.528459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:30.528966 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:30.529624 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:30.529641 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:30.530737 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:30.530742 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:30.531938 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:30.531950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:30.532714 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:30.532723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:30.532981 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:30.533632 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:30.533652 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:30.536163 (+ 2511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:30.539787 (+ 3624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:30.539793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:30.539806 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:30.540187 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:30.540586 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:30.540589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:30.540591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:30.540755 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:30.541393 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:30.541399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:30.541702 (+ 303us) client_negotiation.cc:769] Sending connection context
0927 23:08:30.541907 (+ 205us) client_negotiation.cc:240] Negotiation successful
0927 23:08:30.542316 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":253,"threads_started":1}
I20260927 23:08:30.545394 21360 heartbeater.cc:347] Connected to a master server at 127.3.240.254:40441
I20260927 23:08:30.545740 21360 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:30.546574 21360 heartbeater.cc:511] Master 127.3.240.254:40441 requested a full tablet report, sending...
I20260927 23:08:30.548799 21171 ts_manager.cc:194] Registered new tserver with Master: 66cbbe99ab0f400b846075579d10fa1b (127.3.240.193:37629)
I20260927 23:08:30.550280 21171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:35517
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:08:30.582175 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40441
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:30.893697 21366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:30.893949 21366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:30.894146 21366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:30.901331 21366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:30.901448 21366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:30.901623 21366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:08:30.910950 21366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:30.912155 21366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:30.913391 21366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:30.926013 21372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:30.926003 21374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:30.926004 21371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:30.926874 21366 server_base.cc:1034] running on GCE node
I20260927 23:08:30.927459 21366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:30.928321 21366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:30.929512 21366 hybrid_clock.cc:648] HybridClock initialized: now 1790550510929472 us; error 32 us; skew 500 ppm
I20260927 23:08:30.929814 21366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:30 dist-test-slave-lmfp krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550510, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:30.932818 21366 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:08:30.934063 21366 webserver.cc:460] Webserver started at http://127.3.240.194:33637/ using document root <none> and password file <none>
I20260927 23:08:30.934675 21366 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:30.934769 21366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:30.935063 21366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:30.937692 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "b16e64776fa646698c8083c66bcd2248"
format_stamp: "Formatted at 2026-09-27 23:08:30 on dist-test-slave-lmfp"
server_key: "5866d9e130880e22c23b59b15ddbb81b"
server_key_iv: "4a858bc52d8f87849e65c4cbcd862195"
server_key_version: "encryptionkey@0"
I20260927 23:08:30.938370 21366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "b16e64776fa646698c8083c66bcd2248"
format_stamp: "Formatted at 2026-09-27 23:08:30 on dist-test-slave-lmfp"
server_key: "5866d9e130880e22c23b59b15ddbb81b"
server_key_iv: "4a858bc52d8f87849e65c4cbcd862195"
server_key_version: "encryptionkey@0"
I20260927 23:08:30.943230 21366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260927 23:08:30.946583 21381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:30.947826 21366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:08:30.947985 21366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "b16e64776fa646698c8083c66bcd2248"
format_stamp: "Formatted at 2026-09-27 23:08:30 on dist-test-slave-lmfp"
server_key: "5866d9e130880e22c23b59b15ddbb81b"
server_key_iv: "4a858bc52d8f87849e65c4cbcd862195"
server_key_version: "encryptionkey@0"
I20260927 23:08:30.948163 21366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:30.981742 21366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:30.985468 21366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:30.985781 21366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:30.986716 21366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:30.988035 21366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:30.988131 21366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:30.988270 21366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:30.988349 21366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:31.020166 21366 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:40589
I20260927 23:08:31.020248 21493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:40589 every 8 connection(s)
I20260927 23:08:31.021529 21366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:08:31.023829 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 21366
I20260927 23:08:31.024019 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:08:31.024593 4035 external_mini_cluster.cc:1428] Setting key 724cf3cb1aa22408e811739b77f19231
Sep 27 23:08:31 dist-test-slave-lmfp krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550510, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:31.043694 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.024218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:36521 (local address 127.3.240.254:40441)
0927 23:08:31.024383 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.024388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.025902 (+ 1514us) server_negotiation.cc:407] Connection header received
0927 23:08:31.028211 (+ 2309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.028219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.028314 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.028441 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:31.030302 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.031508 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.033745 (+ 2237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.034472 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.034632 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.037989 (+ 3357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:31.037994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:31.037997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:31.038029 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:31.040651 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:31.041401 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:31.041406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:31.041408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:31.041537 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:31.041953 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:31.041959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:31.041962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:31.042236 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:31.042383 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.043105 (+ 722us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.043331 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:08:31.044803 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.024875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40441 (local address 127.3.240.194:36521)
0927 23:08:31.025723 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:31.025781 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:31.027894 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:31.028670 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:31.028683 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:31.029396 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:31.030113 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.030134 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.031664 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.031671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.033585 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.033601 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.034723 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.034732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.034973 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.035690 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:31.035713 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:31.037774 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:31.040796 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:31.040805 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:31.040822 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:31.041262 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:31.041654 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:31.041659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:31.041663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:31.041846 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:31.042363 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:31.042371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:31.042731 (+ 360us) client_negotiation.cc:769] Sending connection context
0927 23:08:31.042998 (+ 267us) client_negotiation.cc:240] Negotiation successful
0927 23:08:31.043566 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":352,"threads_started":1}
I20260927 23:08:31.046486 21494 heartbeater.cc:347] Connected to a master server at 127.3.240.254:40441
I20260927 23:08:31.046789 21494 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:31.047540 21494 heartbeater.cc:511] Master 127.3.240.254:40441 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260927 23:08:31.049001 21171 ts_manager.cc:194] Registered new tserver with Master: b16e64776fa646698c8083c66bcd2248 (127.3.240.194:40589)
I20260927 23:08:31.049937 21171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:36521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:08:31.077806 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40441
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:31.413259 21499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:31.413512 21499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:31.413601 21499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:31.420632 21499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:31.420727 21499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:31.420846 21499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:08:31.429980 21499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:40441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:31.431239 21499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:31.432485 21499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:31.444787 21504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:31.444763 21505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:31.444960 21507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:31.445475 21499 server_base.cc:1034] running on GCE node
I20260927 23:08:31.446039 21499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:31.446909 21499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:31.448130 21499 hybrid_clock.cc:648] HybridClock initialized: now 1790550511448077 us; error 46 us; skew 500 ppm
I20260927 23:08:31.448415 21499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:31 dist-test-slave-lmfp krb5kdc[21126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550511, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:31.451478 21499 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:08:31.452888 21499 webserver.cc:460] Webserver started at http://127.3.240.195:40541/ using document root <none> and password file <none>
I20260927 23:08:31.453513 21499 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:31.453644 21499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:31.453931 21499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:31.456698 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "06f8fbebafad4e148fbfd446145ab8d8"
format_stamp: "Formatted at 2026-09-27 23:08:31 on dist-test-slave-lmfp"
server_key: "a32e35b7e2f0e6de64ad413afb8969f2"
server_key_iv: "01f049b02cea9bb9053aba7147d2af9f"
server_key_version: "encryptionkey@0"
I20260927 23:08:31.457414 21499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "06f8fbebafad4e148fbfd446145ab8d8"
format_stamp: "Formatted at 2026-09-27 23:08:31 on dist-test-slave-lmfp"
server_key: "a32e35b7e2f0e6de64ad413afb8969f2"
server_key_iv: "01f049b02cea9bb9053aba7147d2af9f"
server_key_version: "encryptionkey@0"
I20260927 23:08:31.462098 21499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 23:08:31.465294 21514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:31.466487 21499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:08:31.466652 21499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "06f8fbebafad4e148fbfd446145ab8d8"
format_stamp: "Formatted at 2026-09-27 23:08:31 on dist-test-slave-lmfp"
server_key: "a32e35b7e2f0e6de64ad413afb8969f2"
server_key_iv: "01f049b02cea9bb9053aba7147d2af9f"
server_key_version: "encryptionkey@0"
I20260927 23:08:31.466845 21499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:31.477311 21499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:31.481143 21499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:31.481415 21499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:31.482357 21499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:31.484444 21499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:31.484540 21499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:31.484659 21499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:31.484730 21499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:31.516790 21499 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:46659
I20260927 23:08:31.516834 21626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:46659 every 8 connection(s)
I20260927 23:08:31.517995 21499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:08:31.523200 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 21499
I20260927 23:08:31.523393 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:08:31.523923 4035 external_mini_cluster.cc:1428] Setting key 89041f9dc8daccf44e876b10d1a343d8
Sep 27 23:08:31 dist-test-slave-lmfp krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550511, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:31.538271 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.520368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:50495 (local address 127.3.240.254:40441)
0927 23:08:31.520580 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.520586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.521915 (+ 1329us) server_negotiation.cc:407] Connection header received
0927 23:08:31.524254 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.524260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.524375 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.524483 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:31.526745 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.527675 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.529298 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.529672 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.529757 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.532961 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:31.532965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:31.532968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:31.532997 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:31.535535 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:31.536365 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:31.536368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:31.536370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:31.536478 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:31.536816 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:31.536819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:31.536821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:31.537051 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:31.537151 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.537743 (+ 592us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.537946 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:08:31.539328 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.520930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40441 (local address 127.3.240.195:50495)
0927 23:08:31.521736 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:31.521780 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:31.523956 (+ 2176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:31.524837 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:31.524847 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:31.525672 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:31.526570 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.526599 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.527827 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.527831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.529177 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.529190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.529856 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.529862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.530073 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.530734 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:31.530754 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:31.532789 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:31.535699 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:31.535705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:31.535727 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:31.536275 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:31.536578 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:31.536581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:31.536583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:31.536734 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:31.537142 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:31.537146 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:31.537401 (+ 255us) client_negotiation.cc:769] Sending connection context
0927 23:08:31.537658 (+ 257us) client_negotiation.cc:240] Negotiation successful
0927 23:08:31.538117 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":348,"threads_started":1}
I20260927 23:08:31.540406 21627 heartbeater.cc:347] Connected to a master server at 127.3.240.254:40441
I20260927 23:08:31.540681 21627 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:31.541337 21627 heartbeater.cc:511] Master 127.3.240.254:40441 requested a full tablet report, sending...
I20260927 23:08:31.542532 21171 ts_manager.cc:194] Registered new tserver with Master: 06f8fbebafad4e148fbfd446145ab8d8 (127.3.240.195:46659)
I20260927 23:08:31.543310 21171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:50495
I20260927 23:08:31.551177 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:08:31.553689 21360 heartbeater.cc:503] Master 127.3.240.254:40441 was elected leader, sending a full tablet report...
I20260927 23:08:31.567070 21362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.555919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.3.240.254:40441)
0927 23:08:31.556049 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.556053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.556537 (+ 484us) server_negotiation.cc:407] Connection header received
0927 23:08:31.556678 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.556682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.556737 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.556810 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:31.558195 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.558864 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.560090 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.560471 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.560569 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.562639 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:31.562644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:31.562647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:31.562674 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:31.565011 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:31.565579 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:31.565583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:31.565585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:31.565704 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:31.565980 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:31.565982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:31.565984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:31.566247 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:31.566417 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.566664 (+ 247us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.566806 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:08:31.570466 21171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:08:31.577373 21171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:08:31.614867 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.589948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.3.240.193:37629)
0927 23:08:31.606193 (+ 16245us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.606204 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.606239 (+ 35us) server_negotiation.cc:407] Connection header received
0927 23:08:31.606366 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.606377 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.606646 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.606922 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:31.607671 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.609248 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.611880 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.613576 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.613718 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.613999 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.614245 (+ 246us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.614436 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16036,"thread_start_us":17222,"threads_started":1}
I20260927 23:08:31.615761 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.589529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37629 (local address 127.0.0.1:43868)
0927 23:08:31.595947 (+ 6418us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:31.595992 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:31.596159 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:31.606941 (+ 10782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:31.606950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:31.606985 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:31.607460 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.607482 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.609465 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.609474 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.611593 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.611620 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.613771 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.613778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.613967 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.614018 (+ 51us) client_negotiation.cc:769] Sending connection context
0927 23:08:31.614145 (+ 127us) client_negotiation.cc:240] Negotiation successful
0927 23:08:31.614320 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6187,"mutex_wait_us":1,"thread_start_us":6025,"threads_started":1}
I20260927 23:08:31.619352 21642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.590038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:40589 (local address 127.0.0.1:41720)
0927 23:08:31.612113 (+ 22075us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:31.612136 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:31.612279 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:31.612929 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:31.612933 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:31.612958 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:31.613325 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.613333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.615077 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.615082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.616683 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.616694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.618404 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.618412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.618620 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.618666 (+ 46us) client_negotiation.cc:769] Sending connection context
0927 23:08:31.618792 (+ 126us) client_negotiation.cc:240] Negotiation successful
0927 23:08:31.618928 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21922,"thread_start_us":22493,"threads_started":1}
I20260927 23:08:31.619414 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.590742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.3.240.194:40589)
0927 23:08:31.606297 (+ 15555us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.606305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.612268 (+ 5963us) server_negotiation.cc:407] Connection header received
0927 23:08:31.612457 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.612467 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.612739 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.612952 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:31.613472 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.614898 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.616818 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.618149 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.618292 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.618451 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.618850 (+ 399us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.619040 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15384,"thread_start_us":15552,"threads_started":1}
I20260927 23:08:31.619813 21295 tablet_service.cc:1467] Processing CreateTablet for tablet e45377f7596c4fdfa13488b5c08280fe (DEFAULT_TABLE table=test-table [id=277feafd32794c9a83096e43c008b4a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:31.621130 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.591056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53566 (local address 127.3.240.195:46659)
0927 23:08:31.611613 (+ 20557us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.611622 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.613139 (+ 1517us) server_negotiation.cc:407] Connection header received
0927 23:08:31.613256 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.613266 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.613504 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.613714 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:31.615201 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.616763 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.618695 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.619979 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.620117 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.620310 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.620650 (+ 340us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.620788 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20347,"thread_start_us":20150,"threads_started":1}
I20260927 23:08:31.621137 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.590492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:46659 (local address 127.0.0.1:53566)
0927 23:08:31.612849 (+ 22357us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:31.612868 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:31.612988 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:31.614056 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:31.614062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:31.614088 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:31.614930 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.614948 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.616986 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.616990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.618514 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.618545 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.620197 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.620205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.620436 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.620488 (+ 52us) client_negotiation.cc:769] Sending connection context
0927 23:08:31.620608 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 23:08:31.620747 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22203,"thread_start_us":21293,"threads_started":1}
I20260927 23:08:31.621411 21295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45377f7596c4fdfa13488b5c08280fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:31.623588 21429 tablet_service.cc:1467] Processing CreateTablet for tablet e45377f7596c4fdfa13488b5c08280fe (DEFAULT_TABLE table=test-table [id=277feafd32794c9a83096e43c008b4a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:31.624925 21429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45377f7596c4fdfa13488b5c08280fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:31.626343 21562 tablet_service.cc:1467] Processing CreateTablet for tablet e45377f7596c4fdfa13488b5c08280fe (DEFAULT_TABLE table=test-table [id=277feafd32794c9a83096e43c008b4a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:31.627638 21562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45377f7596c4fdfa13488b5c08280fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:31.635463 21645 tablet_bootstrap.cc:492] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Bootstrap starting.
I20260927 23:08:31.637451 21646 tablet_bootstrap.cc:492] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Bootstrap starting.
I20260927 23:08:31.640012 21645 tablet_bootstrap.cc:654] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:31.640895 21646 tablet_bootstrap.cc:654] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:31.641402 21645 log.cc:824] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:31.642406 21646 log.cc:824] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:31.644007 21647 tablet_bootstrap.cc:492] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Bootstrap starting.
I20260927 23:08:31.648511 21647 tablet_bootstrap.cc:654] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:31.648725 21645 tablet_bootstrap.cc:492] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: No bootstrap required, opened a new log
I20260927 23:08:31.648967 21645 ts_tablet_manager.cc:1397] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260927 23:08:31.649188 21646 tablet_bootstrap.cc:492] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: No bootstrap required, opened a new log
I20260927 23:08:31.649523 21646 ts_tablet_manager.cc:1397] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20260927 23:08:31.649942 21647 log.cc:824] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:31.653401 21647 tablet_bootstrap.cc:492] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: No bootstrap required, opened a new log
I20260927 23:08:31.653679 21647 ts_tablet_manager.cc:1397] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260927 23:08:31.657730 21645 raft_consensus.cc:348] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.658177 21645 raft_consensus.cc:374] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:31.658279 21645 raft_consensus.cc:729] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66cbbe99ab0f400b846075579d10fa1b, State: Initialized, Role: FOLLOWER
I20260927 23:08:31.658830 21645 consensus_queue.cc:260] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.660480 21645 ts_tablet_manager.cc:1428] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260927 23:08:31.660687 21646 raft_consensus.cc:348] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.661146 21646 raft_consensus.cc:374] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:31.661299 21646 raft_consensus.cc:729] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b16e64776fa646698c8083c66bcd2248, State: Initialized, Role: FOLLOWER
I20260927 23:08:31.661911 21646 consensus_queue.cc:260] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.663071 21494 heartbeater.cc:503] Master 127.3.240.254:40441 was elected leader, sending a full tablet report...
I20260927 23:08:31.663751 21646 ts_tablet_manager.cc:1428] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260927 23:08:31.664294 21647 raft_consensus.cc:348] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.664645 21647 raft_consensus.cc:374] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:31.664728 21647 raft_consensus.cc:729] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f8fbebafad4e148fbfd446145ab8d8, State: Initialized, Role: FOLLOWER
I20260927 23:08:31.665259 21647 consensus_queue.cc:260] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.666247 21627 heartbeater.cc:503] Master 127.3.240.254:40441 was elected leader, sending a full tablet report...
I20260927 23:08:31.667716 21647 ts_tablet_manager.cc:1428] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260927 23:08:31.769620 21628 tablet.cc:2369] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:08:31.773402 21495 tablet.cc:2369] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:08:31.774760 21361 tablet.cc:2369] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:31.887413 21653 raft_consensus.cc:482] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:08:31.887738 21653 raft_consensus.cc:504] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.889345 21653 leader_election.cc:288] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66cbbe99ab0f400b846075579d10fa1b (127.3.240.193:37629), b16e64776fa646698c8083c66bcd2248 (127.3.240.194:40589)
I20260927 23:08:31.894838 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.889970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:41493 (local address 127.3.240.194:40589)
0927 23:08:31.890165 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.890171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.890191 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:08:31.890267 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.890274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.890365 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.890459 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:31.890818 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.891572 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.893237 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.893910 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.894101 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.894191 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.894447 (+ 256us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.894573 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 23:08:31.894862 21629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.889630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:40589 (local address 127.3.240.195:41493)
0927 23:08:31.889869 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:31.889888 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:31.890016 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:31.890494 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:31.890503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:31.890528 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:31.890683 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.890690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.891694 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.891702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.893081 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.893092 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.894080 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.894095 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.894279 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.894301 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 23:08:31.894417 (+ 116us) client_negotiation.cc:240] Negotiation successful
0927 23:08:31.894622 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 23:08:31.895804 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.889756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:50861 (local address 127.3.240.193:37629)
0927 23:08:31.889927 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.889932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.890625 (+ 693us) server_negotiation.cc:407] Connection header received
0927 23:08:31.890799 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.890807 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.890878 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.890967 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:31.891476 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.892310 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.894004 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.895021 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.895197 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.895266 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.895550 (+ 284us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.895642 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 23:08:31.895951 21654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.889839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37629 (local address 127.3.240.195:50861)
0927 23:08:31.890541 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:31.890557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:31.890686 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:31.891009 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:31.891013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:31.891034 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:31.891325 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.891333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.892443 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.892448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.893849 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:31.893860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.895228 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.895234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:31.895373 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.895394 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 23:08:31.895509 (+ 115us) client_negotiation.cc:240] Negotiation successful
0927 23:08:31.895637 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":364,"threads_started":1}
I20260927 23:08:31.895965 21449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45377f7596c4fdfa13488b5c08280fe" candidate_uuid: "06f8fbebafad4e148fbfd446145ab8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b16e64776fa646698c8083c66bcd2248" is_pre_election: true
I20260927 23:08:31.896559 21449 raft_consensus.cc:2454] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f8fbebafad4e148fbfd446145ab8d8 in term 0.
I20260927 23:08:31.896934 21315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45377f7596c4fdfa13488b5c08280fe" candidate_uuid: "06f8fbebafad4e148fbfd446145ab8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cbbe99ab0f400b846075579d10fa1b" is_pre_election: true
I20260927 23:08:31.897217 21518 leader_election.cc:302] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f8fbebafad4e148fbfd446145ab8d8, b16e64776fa646698c8083c66bcd2248; no voters:
I20260927 23:08:31.897502 21315 raft_consensus.cc:2454] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06f8fbebafad4e148fbfd446145ab8d8 in term 0.
I20260927 23:08:31.897603 21653 raft_consensus.cc:2781] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:08:31.897733 21653 raft_consensus.cc:482] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:08:31.897820 21653 raft_consensus.cc:3037] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:31.900067 21653 raft_consensus.cc:504] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.900594 21653 leader_election.cc:288] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [CANDIDATE]: Term 1 election: Requested vote from peers 66cbbe99ab0f400b846075579d10fa1b (127.3.240.193:37629), b16e64776fa646698c8083c66bcd2248 (127.3.240.194:40589)
I20260927 23:08:31.901018 21315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45377f7596c4fdfa13488b5c08280fe" candidate_uuid: "06f8fbebafad4e148fbfd446145ab8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cbbe99ab0f400b846075579d10fa1b"
I20260927 23:08:31.901075 21449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45377f7596c4fdfa13488b5c08280fe" candidate_uuid: "06f8fbebafad4e148fbfd446145ab8d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b16e64776fa646698c8083c66bcd2248"
I20260927 23:08:31.901234 21315 raft_consensus.cc:3037] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:31.901278 21449 raft_consensus.cc:3037] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:31.904304 21449 raft_consensus.cc:2454] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f8fbebafad4e148fbfd446145ab8d8 in term 1.
I20260927 23:08:31.904304 21315 raft_consensus.cc:2454] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06f8fbebafad4e148fbfd446145ab8d8 in term 1.
I20260927 23:08:31.904672 21518 leader_election.cc:302] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f8fbebafad4e148fbfd446145ab8d8, b16e64776fa646698c8083c66bcd2248; no voters:
I20260927 23:08:31.904960 21653 raft_consensus.cc:2781] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:31.905359 21653 raft_consensus.cc:686] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 1 LEADER]: Becoming Leader. State: Replica: 06f8fbebafad4e148fbfd446145ab8d8, State: Running, Role: LEADER
I20260927 23:08:31.905797 21653 consensus_queue.cc:237] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } }
I20260927 23:08:31.910341 21171 catalog_manager.cc:5495] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06f8fbebafad4e148fbfd446145ab8d8 (127.3.240.195). New cstate: current_term: 1 leader_uuid: "06f8fbebafad4e148fbfd446145ab8d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06f8fbebafad4e148fbfd446145ab8d8" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 46659 } health_report { overall_health: HEALTHY } } }
I20260927 23:08:31.966393 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:31.961491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.3.240.195:46659)
0927 23:08:31.961683 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:31.961691 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:31.961711 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:08:31.961802 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:31.961806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:31.961876 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:31.962026 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:08:31.962391 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.963111 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.964525 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:31.964886 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:31.965014 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:31.965438 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:08:31.965881 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:08:31.966027 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:31.966088 (+ 61us) server_negotiation.cc:299] Negotiation successful
0927 23:08:31.966159 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:08:31.973688 21449 raft_consensus.cc:1261] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 1 FOLLOWER]: Refusing update from remote peer 06f8fbebafad4e148fbfd446145ab8d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:31.973688 21315 raft_consensus.cc:1261] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 1 FOLLOWER]: Refusing update from remote peer 06f8fbebafad4e148fbfd446145ab8d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:31.974602 21655 consensus_queue.cc:1035] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b16e64776fa646698c8083c66bcd2248" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:31.974915 21653 consensus_queue.cc:1035] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66cbbe99ab0f400b846075579d10fa1b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:31.984732 21658 mvcc.cc:205] Tried to move back new op lower bound from 7334094897034670080 to 7334094896769744896. Current Snapshot: MvccSnapshot[applied={T|T < 7334094897034670080}]
I20260927 23:08:31.986044 21660 mvcc.cc:205] Tried to move back new op lower bound from 7334094897034670080 to 7334094896769744896. Current Snapshot: MvccSnapshot[applied={T|T < 7334094897034670080}]
I20260927 23:08:31.998922 21170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:31.999279 21170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:32.002777 21170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c490819d6588469a9a93355cbb37988d: Sending DeleteTablet for 3 replicas of tablet e45377f7596c4fdfa13488b5c08280fe
I20260927 23:08:32.004014 21295 tablet_service.cc:1514] Processing DeleteTablet for tablet e45377f7596c4fdfa13488b5c08280fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:31 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:43868
I20260927 23:08:32.003984 21429 tablet_service.cc:1514] Processing DeleteTablet for tablet e45377f7596c4fdfa13488b5c08280fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:31 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41720
I20260927 23:08:32.004165 21562 tablet_service.cc:1514] Processing DeleteTablet for tablet e45377f7596c4fdfa13488b5c08280fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:31 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53566
I20260927 23:08:32.005607 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 21232
I20260927 23:08:32.024338 21668 tablet_replica.cc:330] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: stopping tablet replica
I20260927 23:08:32.025277 21668 raft_consensus.cc:2229] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:32.025868 21668 raft_consensus.cc:2258] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:32.035660 21667 tablet_replica.cc:330] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: stopping tablet replica
I20260927 23:08:32.036103 21667 raft_consensus.cc:2229] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:32.036655 21667 raft_consensus.cc:2258] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:32.037377 21668 ts_tablet_manager.cc:1905] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:32.039937 21667 ts_tablet_manager.cc:1905] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:32.040700 21669 tablet_replica.cc:330] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: stopping tablet replica
I20260927 23:08:32.043133 21669 raft_consensus.cc:2229] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:08:32.044454 21668 ts_tablet_manager.cc:1918] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:32.044662 21668 log.cc:1196] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/e45377f7596c4fdfa13488b5c08280fe
I20260927 23:08:32.045264 21668 ts_tablet_manager.cc:1939] T e45377f7596c4fdfa13488b5c08280fe P b16e64776fa646698c8083c66bcd2248: Deleting consensus metadata
I20260927 23:08:32.046770 21667 ts_tablet_manager.cc:1918] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:32.046917 21667 log.cc:1196] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/e45377f7596c4fdfa13488b5c08280fe
I20260927 23:08:32.047147 21158 catalog_manager.cc:4841] TS b16e64776fa646698c8083c66bcd2248 (127.3.240.194:40589): tablet e45377f7596c4fdfa13488b5c08280fe (table test-table [id=277feafd32794c9a83096e43c008b4a2]) successfully deleted
I20260927 23:08:32.047494 21667 ts_tablet_manager.cc:1939] T e45377f7596c4fdfa13488b5c08280fe P 66cbbe99ab0f400b846075579d10fa1b: Deleting consensus metadata
I20260927 23:08:32.048014 21669 raft_consensus.cc:2258] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 23:08:32 dist-test-slave-lmfp krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:08:32.049125 21156 catalog_manager.cc:4841] TS 66cbbe99ab0f400b846075579d10fa1b (127.3.240.193:37629): tablet e45377f7596c4fdfa13488b5c08280fe (table test-table [id=277feafd32794c9a83096e43c008b4a2]) successfully deleted
I20260927 23:08:32.050052 21669 ts_tablet_manager.cc:1905] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:32.055039 21669 ts_tablet_manager.cc:1918] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:32.055202 21669 log.cc:1196] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/e45377f7596c4fdfa13488b5c08280fe
I20260927 23:08:32.055719 21669 ts_tablet_manager.cc:1939] T e45377f7596c4fdfa13488b5c08280fe P 06f8fbebafad4e148fbfd446145ab8d8: Deleting consensus metadata
I20260927 23:08:32.057610 21155 catalog_manager.cc:4841] TS 06f8fbebafad4e148fbfd446145ab8d8 (127.3.240.195:46659): tablet e45377f7596c4fdfa13488b5c08280fe (table test-table [id=277feafd32794c9a83096e43c008b4a2]) successfully deleted
I20260927 23:08:32.062436 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:32.007382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43872 (local address 127.3.240.193:37629)
0927 23:08:32.007630 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:32.007640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:32.017589 (+ 9949us) server_negotiation.cc:407] Connection header received
0927 23:08:32.018067 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:32.018076 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:32.018190 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:32.018278 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:32.036544 (+ 18266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:32.038074 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:32.042080 (+ 4006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:32.042762 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:32.042851 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:32.056446 (+ 13595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:32.056459 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:32.056475 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:32.056515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:32.059688 (+ 3173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:32.060364 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:32.060370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:32.060372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:32.060498 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:32.060888 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:32.060893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:32.060895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:32.061235 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:32.061531 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:32.061827 (+ 296us) server_negotiation.cc:299] Negotiation successful
0927 23:08:32.062063 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:08:32.063128 21355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:32.148262 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 21232
I20260927 23:08:32.166028 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 21366
Sep 27 23:08:32 dist-test-slave-lmfp krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:08:32.181659 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:32.166804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.3.240.194:40589)
0927 23:08:32.166999 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:32.167004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:32.167052 (+ 48us) server_negotiation.cc:407] Connection header received
0927 23:08:32.167218 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:32.167223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:32.167299 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:32.167394 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:32.168743 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:32.169428 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:32.170723 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:32.171432 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:32.171574 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:32.175456 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:32.175475 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:32.175496 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:32.175537 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:32.179078 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:32.179672 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:32.179683 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:32.179686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:32.179838 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:32.180165 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:32.180170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:32.180173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:32.180614 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:32.180894 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:32.181057 (+ 163us) server_negotiation.cc:299] Negotiation successful
0927 23:08:32.181279 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 23:08:32.182322 21489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:32.268074 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 21366
I20260927 23:08:32.285288 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 21499
Sep 27 23:08:32 dist-test-slave-lmfp krb5kdc[21126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550509, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:08:32.303357 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:32.286179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.3.240.195:46659)
0927 23:08:32.286381 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:32.286385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:32.286439 (+ 54us) server_negotiation.cc:407] Connection header received
0927 23:08:32.286632 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:32.286636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:32.286702 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:32.286794 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:32.288498 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:32.289306 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:32.291250 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:32.291716 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:32.291802 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:32.296855 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:32.296871 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:32.296893 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:32.296936 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:32.300567 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:32.301195 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:32.301206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:32.301209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:32.301401 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:32.301753 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:32.301758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:32.301761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:32.302184 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:32.302516 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:32.302710 (+ 194us) server_negotiation.cc:299] Negotiation successful
0927 23:08:32.302913 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 23:08:32.304051 21622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:32.422075 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 21499
I20260927 23:08:32.441780 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 21139
I20260927 23:08:32.442842 21201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:32.550249 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 21139
2026-09-27T23:08:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4347 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-27T23:08:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:32Z Disabled control of system clock
I20260927 23:08:32.621153 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46015
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 23:08:32.934144 21678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:32.934422 21678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:32.934507 21678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:32.941699 21678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:32.941810 21678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:32.941852 21678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:32.941915 21678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:32.941964 21678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:32.951021 21678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:32.952270 21678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:32.953981 21678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:32.965602 21686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:32.965677 21683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:32.965705 21684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:32.966530 21678 server_base.cc:1034] running on GCE node
I20260927 23:08:32.967265 21678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:32.968528 21678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:32.969765 21678 hybrid_clock.cc:648] HybridClock initialized: now 1790550512969703 us; error 67 us; skew 500 ppm
I20260927 23:08:32.970112 21678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:32.977243 21678 webserver.cc:460] Webserver started at http://127.3.240.254:38763/ using document root <none> and password file <none>
I20260927 23:08:32.977876 21678 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:32.978012 21678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:32.978314 21678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:32.980955 21678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "dc25005c181346d6890222d9ec9aa163"
format_stamp: "Formatted at 2026-09-27 23:08:32 on dist-test-slave-lmfp"
server_key: "56506c05d5d89f2c65bb1fff237a2b7a"
server_key_iv: "9942432152be8c5d583cc5ec89f4f2ae"
server_key_version: "encryptionkey@0"
I20260927 23:08:32.981724 21678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "dc25005c181346d6890222d9ec9aa163"
format_stamp: "Formatted at 2026-09-27 23:08:32 on dist-test-slave-lmfp"
server_key: "56506c05d5d89f2c65bb1fff237a2b7a"
server_key_iv: "9942432152be8c5d583cc5ec89f4f2ae"
server_key_version: "encryptionkey@0"
I20260927 23:08:32.986548 21678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 23:08:32.989838 21692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:32.991195 21678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:08:32.991355 21678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "dc25005c181346d6890222d9ec9aa163"
format_stamp: "Formatted at 2026-09-27 23:08:32 on dist-test-slave-lmfp"
server_key: "56506c05d5d89f2c65bb1fff237a2b7a"
server_key_iv: "9942432152be8c5d583cc5ec89f4f2ae"
server_key_version: "encryptionkey@0"
I20260927 23:08:32.991581 21678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:33.009219 21678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:33.009996 21678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:33.010259 21678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:33.027534 21678 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46015
I20260927 23:08:33.027560 21743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46015 every 8 connection(s)
I20260927 23:08:33.028756 21678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:33.029414 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 21678
I20260927 23:08:33.029633 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:33.030179 4035 external_mini_cluster.cc:1428] Setting key 7c7a462ffff2b5064f9135d509500150
I20260927 23:08:33.032886 21744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:33.046046 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:33.032246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.3.240.254:46015)
0927 23:08:33.033397 (+ 1151us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:33.033412 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:33.033517 (+ 105us) server_negotiation.cc:407] Connection header received
0927 23:08:33.035829 (+ 2312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:33.035855 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:33.036469 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:33.036952 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:33.037523 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.039616 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.041066 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.042086 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.042271 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:33.043356 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:33.043366 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:33.043383 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:33.044112 (+ 729us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:33.044200 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:33.044226 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:33.044349 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:33.044522 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 23:08:33.044853 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":414,"threads_started":1}
I20260927 23:08:33.047199 21744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163: Bootstrap starting.
I20260927 23:08:33.050117 21744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:33.051064 21744 log.cc:824] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:33.053465 21744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163: No bootstrap required, opened a new log
I20260927 23:08:33.060242 21744 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25005c181346d6890222d9ec9aa163" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46015 } }
I20260927 23:08:33.060572 21744 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:33.060675 21744 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc25005c181346d6890222d9ec9aa163, State: Initialized, Role: FOLLOWER
I20260927 23:08:33.061172 21744 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25005c181346d6890222d9ec9aa163" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46015 } }
I20260927 23:08:33.061408 21744 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:33.061556 21744 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:33.061718 21744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:33.063783 21744 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25005c181346d6890222d9ec9aa163" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46015 } }
I20260927 23:08:33.064254 21744 leader_election.cc:302] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc25005c181346d6890222d9ec9aa163; no voters:
I20260927 23:08:33.064823 21744 leader_election.cc:288] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:33.064954 21749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:33.065723 21749 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [term 1 LEADER]: Becoming Leader. State: Replica: dc25005c181346d6890222d9ec9aa163, State: Running, Role: LEADER
I20260927 23:08:33.066321 21749 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25005c181346d6890222d9ec9aa163" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46015 } }
I20260927 23:08:33.066895 21744 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:33.069023 21751 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc25005c181346d6890222d9ec9aa163. Latest consensus state: current_term: 1 leader_uuid: "dc25005c181346d6890222d9ec9aa163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25005c181346d6890222d9ec9aa163" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46015 } } }
I20260927 23:08:33.069051 21750 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc25005c181346d6890222d9ec9aa163" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc25005c181346d6890222d9ec9aa163" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46015 } } }
I20260927 23:08:33.069628 21751 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:33.069842 21750 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:33.072016 21754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:33.077062 21754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:33.084630 21754 catalog_manager.cc:1347] Generated new cluster ID: ea7ca65ca87841d5940f6e90c43fff2e
I20260927 23:08:33.084744 21754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:33.102203 21754 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:33.103806 21754 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:33.113813 21754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163: Generated new TSK 0
I20260927 23:08:33.114598 21754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:33.127931 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46015
--builtin_ntp_servers=127.3.240.212:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 23:08:33.441876 21768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:33.442130 21768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:33.442291 21768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:33.449455 21768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:33.449548 21768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:33.449713 21768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:08:33.458233 21768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:33.459452 21768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:33.461100 21768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:33.473433 21773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:33.473469 21776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:33.473513 21774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:33.474525 21768 server_base.cc:1034] running on GCE node
I20260927 23:08:33.475165 21768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:33.476070 21768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:33.477257 21768 hybrid_clock.cc:648] HybridClock initialized: now 1790550513477218 us; error 28 us; skew 500 ppm
I20260927 23:08:33.477550 21768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:33.479748 21768 webserver.cc:460] Webserver started at http://127.3.240.193:40763/ using document root <none> and password file <none>
I20260927 23:08:33.480376 21768 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:33.480495 21768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:33.480795 21768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:33.483568 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "bc9b2d40233f4c9ab9545fb6184b0852"
format_stamp: "Formatted at 2026-09-27 23:08:33 on dist-test-slave-lmfp"
server_key: "b86bda38d300b377c78a1a75e79f44d9"
server_key_iv: "1e5155f4b240ca8ec9e17b013a4e408a"
server_key_version: "encryptionkey@0"
I20260927 23:08:33.484287 21768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "bc9b2d40233f4c9ab9545fb6184b0852"
format_stamp: "Formatted at 2026-09-27 23:08:33 on dist-test-slave-lmfp"
server_key: "b86bda38d300b377c78a1a75e79f44d9"
server_key_iv: "1e5155f4b240ca8ec9e17b013a4e408a"
server_key_version: "encryptionkey@0"
I20260927 23:08:33.489024 21768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260927 23:08:33.492455 21782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:33.493695 21768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260927 23:08:33.493848 21768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "bc9b2d40233f4c9ab9545fb6184b0852"
format_stamp: "Formatted at 2026-09-27 23:08:33 on dist-test-slave-lmfp"
server_key: "b86bda38d300b377c78a1a75e79f44d9"
server_key_iv: "1e5155f4b240ca8ec9e17b013a4e408a"
server_key_version: "encryptionkey@0"
I20260927 23:08:33.494062 21768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:33.506064 21768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:33.506757 21768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:33.507083 21768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:33.507956 21768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:33.509359 21768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:33.509464 21768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:33.509586 21768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:33.509680 21768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:33.540871 21768 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:44383
I20260927 23:08:33.540939 21894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:44383 every 8 connection(s)
I20260927 23:08:33.542260 21768 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:08:33.549729 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 21768
I20260927 23:08:33.549875 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:08:33.550323 4035 external_mini_cluster.cc:1428] Setting key 9241f012f92a995deda0305fcdb56ef3
I20260927 23:08:33.552991 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46015
--builtin_ntp_servers=127.3.240.212:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 23:08:33.554621 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:33.544827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:38127 (local address 127.3.240.254:46015)
0927 23:08:33.545044 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:33.545050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:33.546450 (+ 1400us) server_negotiation.cc:407] Connection header received
0927 23:08:33.548178 (+ 1728us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:33.548182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:33.548244 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:33.548321 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:33.549335 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.550147 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.551903 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.552390 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.552514 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:33.553845 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:33.553849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:33.553853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:33.553868 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:33.553897 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:33.553908 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:33.553997 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:33.554220 (+ 223us) server_negotiation.cc:299] Negotiation successful
0927 23:08:33.554311 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 23:08:33.555595 21897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:33.545258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46015 (local address 127.3.240.193:38127)
0927 23:08:33.546271 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:33.546314 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:33.547925 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:33.548507 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:33.548514 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:33.548615 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:33.549161 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:33.549176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.550311 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.550316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:33.551755 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:33.551769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.552606 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.552613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:33.552854 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:33.553495 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:33.553498 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:33.553539 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:33.553626 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:33.553997 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:33.554010 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:33.554020 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 23:08:33.554200 (+ 180us) client_negotiation.cc:240] Negotiation successful
0927 23:08:33.554547 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":342,"threads_started":1}
I20260927 23:08:33.556977 21895 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46015
I20260927 23:08:33.557264 21895 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:33.557919 21895 heartbeater.cc:511] Master 127.3.240.254:46015 requested a full tablet report, sending...
I20260927 23:08:33.559942 21709 ts_manager.cc:194] Registered new tserver with Master: bc9b2d40233f4c9ab9545fb6184b0852 (127.3.240.193:44383)
I20260927 23:08:33.561381 21709 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.193:38127
W20260927 23:08:33.861554 21898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:33.861804 21898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:33.861944 21898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:33.869026 21898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:33.869113 21898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:33.869279 21898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:08:33.877957 21898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:33.879155 21898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:33.880825 21898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:33.893106 21903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:33.893111 21904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:33.893347 21906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:33.893888 21898 server_base.cc:1034] running on GCE node
I20260927 23:08:33.894513 21898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:33.895509 21898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:33.896725 21898 hybrid_clock.cc:648] HybridClock initialized: now 1790550513896691 us; error 53 us; skew 500 ppm
I20260927 23:08:33.897024 21898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:33.899370 21898 webserver.cc:460] Webserver started at http://127.3.240.194:43751/ using document root <none> and password file <none>
I20260927 23:08:33.900020 21898 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:33.900170 21898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:33.900476 21898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:33.903330 21898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "b88194a8533f4d1e977904b2e7180d34"
format_stamp: "Formatted at 2026-09-27 23:08:33 on dist-test-slave-lmfp"
server_key: "82879e9241c845b49218f4d7206abe66"
server_key_iv: "dd726970d844e332634b05fffd1daefb"
server_key_version: "encryptionkey@0"
I20260927 23:08:33.904066 21898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "b88194a8533f4d1e977904b2e7180d34"
format_stamp: "Formatted at 2026-09-27 23:08:33 on dist-test-slave-lmfp"
server_key: "82879e9241c845b49218f4d7206abe66"
server_key_iv: "dd726970d844e332634b05fffd1daefb"
server_key_version: "encryptionkey@0"
I20260927 23:08:33.908574 21898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:08:33.911754 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:33.912876 21898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:08:33.913030 21898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "b88194a8533f4d1e977904b2e7180d34"
format_stamp: "Formatted at 2026-09-27 23:08:33 on dist-test-slave-lmfp"
server_key: "82879e9241c845b49218f4d7206abe66"
server_key_iv: "dd726970d844e332634b05fffd1daefb"
server_key_version: "encryptionkey@0"
I20260927 23:08:33.913235 21898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:33.940122 21898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:33.940835 21898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:33.941112 21898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:33.941992 21898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:33.943414 21898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:33.943523 21898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:33.943650 21898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:33.943730 21898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:33.975059 21898 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:45405
I20260927 23:08:33.975098 22024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:45405 every 8 connection(s)
I20260927 23:08:33.976274 21898 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:08:33.985828 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 21898
I20260927 23:08:33.985981 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:08:33.986397 4035 external_mini_cluster.cc:1428] Setting key a8adb4b86be26f9eb832defd0a40944c
I20260927 23:08:33.987579 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:33.978623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:57375 (local address 127.3.240.254:46015)
0927 23:08:33.978787 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:33.978792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:33.980166 (+ 1374us) server_negotiation.cc:407] Connection header received
0927 23:08:33.981801 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:33.981805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:33.981869 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:33.981953 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:33.982848 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.983585 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.984989 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.985370 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.985458 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:33.986735 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:33.986740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:33.986744 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:33.986764 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:33.986798 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:33.986812 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:33.986899 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:33.987170 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 23:08:33.987266 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:08:33.988660 22027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:33.979184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46015 (local address 127.3.240.194:57375)
0927 23:08:33.980019 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:33.980068 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:33.981584 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:33.982128 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:33.982135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:33.982213 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:33.982714 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:33.982732 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.983711 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.983715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:33.984870 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:33.984881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:33.985526 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:33.985530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:33.985710 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:33.986399 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:33.986401 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:33.986440 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:33.986534 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:33.986896 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:33.986902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:33.986907 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 23:08:33.987147 (+ 240us) client_negotiation.cc:240] Negotiation successful
0927 23:08:33.987469 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":278,"threads_started":1}
I20260927 23:08:33.989400 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46015
--builtin_ntp_servers=127.3.240.212:33291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 23:08:33.989913 22025 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46015
I20260927 23:08:33.990281 22025 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:33.991159 22025 heartbeater.cc:511] Master 127.3.240.254:46015 requested a full tablet report, sending...
I20260927 23:08:33.992607 21709 ts_manager.cc:194] Registered new tserver with Master: b88194a8533f4d1e977904b2e7180d34 (127.3.240.194:45405)
I20260927 23:08:33.993360 21709 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.194:57375
W20260927 23:08:34.308641 22028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:34.308956 22028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:34.309118 22028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:34.316195 22028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:34.316315 22028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:34.316427 22028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:08:34.325227 22028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:34.326387 22028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:34.328148 22028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:34.340651 22034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:34.340736 22036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:34.340682 22033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:34.342113 22028 server_base.cc:1034] running on GCE node
I20260927 23:08:34.342727 22028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:34.343688 22028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:34.344933 22028 hybrid_clock.cc:648] HybridClock initialized: now 1790550514344885 us; error 53 us; skew 500 ppm
I20260927 23:08:34.345274 22028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:34.347527 22028 webserver.cc:460] Webserver started at http://127.3.240.195:40639/ using document root <none> and password file <none>
I20260927 23:08:34.348199 22028 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:34.348325 22028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:34.348635 22028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:34.351468 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "187f342bdefd4b27a41fa3da3aaea0ad"
format_stamp: "Formatted at 2026-09-27 23:08:34 on dist-test-slave-lmfp"
server_key: "46462dff6bb6bc38522d60d938653a9f"
server_key_iv: "c006ca9a537629a1babd49f91569333f"
server_key_version: "encryptionkey@0"
I20260927 23:08:34.352213 22028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "187f342bdefd4b27a41fa3da3aaea0ad"
format_stamp: "Formatted at 2026-09-27 23:08:34 on dist-test-slave-lmfp"
server_key: "46462dff6bb6bc38522d60d938653a9f"
server_key_iv: "c006ca9a537629a1babd49f91569333f"
server_key_version: "encryptionkey@0"
I20260927 23:08:34.356976 22028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:08:34.360589 22042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:34.361867 22028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:08:34.362023 22028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "187f342bdefd4b27a41fa3da3aaea0ad"
format_stamp: "Formatted at 2026-09-27 23:08:34 on dist-test-slave-lmfp"
server_key: "46462dff6bb6bc38522d60d938653a9f"
server_key_iv: "c006ca9a537629a1babd49f91569333f"
server_key_version: "encryptionkey@0"
I20260927 23:08:34.362242 22028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:34.380790 22028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:34.381491 22028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:34.381816 22028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:34.382680 22028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:34.383975 22028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:34.384102 22028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:34.384217 22028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:34.384298 22028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:34.414476 22028 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:43931
I20260927 23:08:34.414564 22154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:43931 every 8 connection(s)
I20260927 23:08:34.415681 22028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:08:34.422755 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 22028
I20260927 23:08:34.422919 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:08:34.423432 4035 external_mini_cluster.cc:1428] Setting key 6c6c07d5419c961278074af3124f10b5
I20260927 23:08:34.428047 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.418324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:40983 (local address 127.3.240.254:46015)
0927 23:08:34.418492 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.418496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.420036 (+ 1540us) server_negotiation.cc:407] Connection header received
0927 23:08:34.421713 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.421717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.421796 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.421871 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:34.423171 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.423923 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.425450 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.425961 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.426080 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.427402 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:34.427407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:34.427410 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:34.427425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:34.427455 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:34.427467 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:34.427542 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.427748 (+ 206us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.427832 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:08:34.428934 22157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.418837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46015 (local address 127.3.240.195:40983)
0927 23:08:34.419878 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.419919 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:34.421469 (+ 1550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:34.422062 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:34.422069 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:34.422161 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:34.422946 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.422973 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.424096 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.424100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.425304 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.425316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.426186 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.426192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.426404 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.427126 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:34.427129 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:34.427171 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:34.427247 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:34.427548 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:34.427562 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:34.427567 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 23:08:34.427716 (+ 149us) client_negotiation.cc:240] Negotiation successful
0927 23:08:34.427991 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":349,"threads_started":1}
I20260927 23:08:34.430056 22155 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46015
I20260927 23:08:34.430333 22155 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:34.431082 22155 heartbeater.cc:511] Master 127.3.240.254:46015 requested a full tablet report, sending...
I20260927 23:08:34.432375 21709 ts_manager.cc:194] Registered new tserver with Master: 187f342bdefd4b27a41fa3da3aaea0ad (127.3.240.195:43931)
I20260927 23:08:34.433131 21709 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.195:40983
I20260927 23:08:34.438280 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:08:34.447867 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.442445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.3.240.254:46015)
0927 23:08:34.442682 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.442691 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.443116 (+ 425us) server_negotiation.cc:407] Connection header received
0927 23:08:34.443262 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.443266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.443362 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.443455 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:34.443902 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.444587 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.445840 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.446222 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.446313 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.447282 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:34.447287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:34.447290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:34.447306 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:34.447336 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:34.447347 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:34.447421 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.447551 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.447618 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 23:08:34.451301 21709 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:08:34.457849 21709 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:08:34.490427 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.469685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57224 (local address 127.3.240.193:44383)
0927 23:08:34.480771 (+ 11086us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.480784 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.480825 (+ 41us) server_negotiation.cc:407] Connection header received
0927 23:08:34.480962 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.480974 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.481345 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.482259 (+ 914us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:34.483471 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.485019 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.487805 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.488963 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.489113 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.489767 (+ 654us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.489841 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.490018 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10864,"thread_start_us":10723,"threads_started":1}
I20260927 23:08:34.490535 22167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.469207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44383 (local address 127.0.0.1:57224)
0927 23:08:34.470008 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.470058 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:34.470226 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:34.482063 (+ 11837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:34.482071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:34.482893 (+ 822us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:34.483255 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.483268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.485162 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.485167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.487616 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.487628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.489183 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.489190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.489565 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.489640 (+ 75us) client_negotiation.cc:769] Sending connection context
0927 23:08:34.489769 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 23:08:34.489986 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":158,"spinlock_wait_cycles":1769344,"thread_start_us":967,"threads_started":1}
I20260927 23:08:34.490674 22171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.469467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:45405 (local address 127.0.0.1:35036)
0927 23:08:34.481118 (+ 11651us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.481146 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:34.482700 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:34.483486 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:34.483492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:34.483518 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:34.483817 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.483825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.485692 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.485698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.487941 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.487952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.489862 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.489869 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.490075 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.490108 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 23:08:34.490186 (+ 78us) client_negotiation.cc:240] Negotiation successful
0927 23:08:34.490319 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11483,"mutex_wait_us":70,"thread_start_us":12251,"threads_started":1}
I20260927 23:08:34.491784 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.469848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35036 (local address 127.3.240.194:45405)
0927 23:08:34.482884 (+ 13036us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.482892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.482918 (+ 26us) server_negotiation.cc:407] Connection header received
0927 23:08:34.483061 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.483068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.483297 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.483963 (+ 666us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:34.484268 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.485539 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.488149 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.489669 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.491010 (+ 1341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.491231 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.491302 (+ 71us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.491457 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12897,"thread_start_us":12781,"threads_started":1}
I20260927 23:08:34.492856 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.470767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43931 (local address 127.0.0.1:51314)
0927 23:08:34.481984 (+ 11217us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.482004 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:34.483467 (+ 1463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:34.484090 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:34.484094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:34.484116 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:34.484465 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.484477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.486135 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.486145 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.489461 (+ 3316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.489481 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.492003 (+ 2522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.492012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.492197 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.492228 (+ 31us) client_negotiation.cc:769] Sending connection context
0927 23:08:34.492322 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 23:08:34.492431 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11096,"mutex_wait_us":31,"spinlock_wait_cycles":1750784,"thread_start_us":10013,"threads_started":1}
I20260927 23:08:34.493744 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.471445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51314 (local address 127.3.240.195:43931)
0927 23:08:34.483402 (+ 11957us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.483411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.483444 (+ 33us) server_negotiation.cc:407] Connection header received
0927 23:08:34.483634 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.483644 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.483880 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.484088 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:34.484674 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.485989 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.489689 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.491377 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.491497 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.491766 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.493334 (+ 1568us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.493480 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11755,"thread_start_us":11594,"threads_started":1}
I20260927 23:08:34.494882 21830 tablet_service.cc:1467] Processing CreateTablet for tablet 7e09f90c3e9646669b2b53e894dd74e4 (DEFAULT_TABLE table=test-table [id=f81030ef1f264e28bb39db26549388a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:34.495503 21960 tablet_service.cc:1467] Processing CreateTablet for tablet 7e09f90c3e9646669b2b53e894dd74e4 (DEFAULT_TABLE table=test-table [id=f81030ef1f264e28bb39db26549388a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:34.496505 21830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e09f90c3e9646669b2b53e894dd74e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:34.496847 21960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e09f90c3e9646669b2b53e894dd74e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:34.496626 22090 tablet_service.cc:1467] Processing CreateTablet for tablet 7e09f90c3e9646669b2b53e894dd74e4 (DEFAULT_TABLE table=test-table [id=f81030ef1f264e28bb39db26549388a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:34.497728 22090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e09f90c3e9646669b2b53e894dd74e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:34.508492 22173 tablet_bootstrap.cc:492] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Bootstrap starting.
I20260927 23:08:34.511430 22174 tablet_bootstrap.cc:492] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Bootstrap starting.
I20260927 23:08:34.512620 22173 tablet_bootstrap.cc:654] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:34.513311 22175 tablet_bootstrap.cc:492] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: Bootstrap starting.
I20260927 23:08:34.514353 22173 log.cc:824] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:34.514964 22174 tablet_bootstrap.cc:654] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:34.516541 22174 log.cc:824] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:34.516856 22175 tablet_bootstrap.cc:654] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:34.518136 22175 log.cc:824] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:34.519984 22174 tablet_bootstrap.cc:492] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: No bootstrap required, opened a new log
I20260927 23:08:34.520310 22174 ts_tablet_manager.cc:1397] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260927 23:08:34.521296 22175 tablet_bootstrap.cc:492] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: No bootstrap required, opened a new log
I20260927 23:08:34.521582 22175 ts_tablet_manager.cc:1397] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 23:08:34.522253 22173 tablet_bootstrap.cc:492] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: No bootstrap required, opened a new log
I20260927 23:08:34.522485 22173 ts_tablet_manager.cc:1397] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260927 23:08:34.528527 22175 raft_consensus.cc:348] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.528899 22175 raft_consensus.cc:374] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:34.529032 22175 raft_consensus.cc:729] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9b2d40233f4c9ab9545fb6184b0852, State: Initialized, Role: FOLLOWER
I20260927 23:08:34.529664 22175 consensus_queue.cc:260] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.530488 22174 raft_consensus.cc:348] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.530848 21895 heartbeater.cc:503] Master 127.3.240.254:46015 was elected leader, sending a full tablet report...
I20260927 23:08:34.530889 22174 raft_consensus.cc:374] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:34.530973 22174 raft_consensus.cc:729] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b88194a8533f4d1e977904b2e7180d34, State: Initialized, Role: FOLLOWER
I20260927 23:08:34.531490 22175 ts_tablet_manager.cc:1428] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260927 23:08:34.531536 22174 consensus_queue.cc:260] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.532581 22025 heartbeater.cc:503] Master 127.3.240.254:46015 was elected leader, sending a full tablet report...
I20260927 23:08:34.533185 22174 ts_tablet_manager.cc:1428] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260927 23:08:34.535388 22173 raft_consensus.cc:348] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.535828 22173 raft_consensus.cc:374] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:34.535960 22173 raft_consensus.cc:729] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 187f342bdefd4b27a41fa3da3aaea0ad, State: Initialized, Role: FOLLOWER
I20260927 23:08:34.536521 22173 consensus_queue.cc:260] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.537603 22155 heartbeater.cc:503] Master 127.3.240.254:46015 was elected leader, sending a full tablet report...
I20260927 23:08:34.538198 22173 ts_tablet_manager.cc:1428] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Time spent starting tablet: real 0.016s user 0.005s sys 0.008s
W20260927 23:08:34.544369 21896 tablet.cc:2369] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:34.662348 22179 raft_consensus.cc:482] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:08:34.662660 22179 raft_consensus.cc:504] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.664160 22179 leader_election.cc:288] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b88194a8533f4d1e977904b2e7180d34 (127.3.240.194:45405), 187f342bdefd4b27a41fa3da3aaea0ad (127.3.240.195:43931)
W20260927 23:08:34.667299 22156 tablet.cc:2369] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:34.671247 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.664554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:42629 (local address 127.3.240.195:43931)
0927 23:08:34.664704 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.664709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.665362 (+ 653us) server_negotiation.cc:407] Connection header received
0927 23:08:34.665542 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.665546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.665613 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.665692 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:34.666506 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.667583 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.669557 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.670344 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.670511 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.670582 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.670849 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.670965 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 23:08:34.671339 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.664544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:41409 (local address 127.3.240.194:45405)
0927 23:08:34.664704 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.664709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.665301 (+ 592us) server_negotiation.cc:407] Connection header received
0927 23:08:34.665498 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.665504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.665589 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.665691 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:34.666443 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.667466 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.669669 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.670437 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.670610 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.670675 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.670942 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.671095 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:08:34.671422 22182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.664526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43931 (local address 127.3.240.193:42629)
0927 23:08:34.665247 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.665260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:34.665416 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:34.665752 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:34.665756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:34.665806 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:34.666359 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.666370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.667700 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.667706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.669368 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.669379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.670516 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.670522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.670699 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.670722 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 23:08:34.670824 (+ 102us) client_negotiation.cc:240] Negotiation successful
0927 23:08:34.670945 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":173,"spinlock_wait_cycles":6144,"thread_start_us":329,"threads_started":1}
I20260927 23:08:34.671458 22183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.664378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:45405 (local address 127.3.240.193:41409)
0927 23:08:34.665186 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.665203 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:34.665364 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:34.665752 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:34.665756 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:34.665805 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:34.666280 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.666299 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.667620 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.667625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.669370 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:34.669382 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.670625 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.670631 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:34.670790 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.670811 (+ 21us) client_negotiation.cc:769] Sending connection context
0927 23:08:34.670904 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 23:08:34.671046 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":155,"spinlock_wait_cycles":25088,"thread_start_us":490,"threads_started":1}
I20260927 23:08:34.672369 22110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e09f90c3e9646669b2b53e894dd74e4" candidate_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" is_pre_election: true
I20260927 23:08:34.672540 21980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e09f90c3e9646669b2b53e894dd74e4" candidate_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88194a8533f4d1e977904b2e7180d34" is_pre_election: true
I20260927 23:08:34.672889 22110 raft_consensus.cc:2454] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc9b2d40233f4c9ab9545fb6184b0852 in term 0.
I20260927 23:08:34.672991 21980 raft_consensus.cc:2454] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc9b2d40233f4c9ab9545fb6184b0852 in term 0.
I20260927 23:08:34.673556 21786 leader_election.cc:302] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 187f342bdefd4b27a41fa3da3aaea0ad, bc9b2d40233f4c9ab9545fb6184b0852; no voters:
I20260927 23:08:34.673945 22179 raft_consensus.cc:2781] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:08:34.674093 22179 raft_consensus.cc:482] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:08:34.674188 22179 raft_consensus.cc:3037] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:34.676285 22179 raft_consensus.cc:504] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.676872 22179 leader_election.cc:288] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [CANDIDATE]: Term 1 election: Requested vote from peers b88194a8533f4d1e977904b2e7180d34 (127.3.240.194:45405), 187f342bdefd4b27a41fa3da3aaea0ad (127.3.240.195:43931)
I20260927 23:08:34.677223 21980 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e09f90c3e9646669b2b53e894dd74e4" candidate_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b88194a8533f4d1e977904b2e7180d34"
I20260927 23:08:34.677292 22110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e09f90c3e9646669b2b53e894dd74e4" candidate_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "187f342bdefd4b27a41fa3da3aaea0ad"
I20260927 23:08:34.677408 21980 raft_consensus.cc:3037] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:34.677456 22110 raft_consensus.cc:3037] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:34.680370 21980 raft_consensus.cc:2454] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc9b2d40233f4c9ab9545fb6184b0852 in term 1.
I20260927 23:08:34.680368 22110 raft_consensus.cc:2454] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc9b2d40233f4c9ab9545fb6184b0852 in term 1.
I20260927 23:08:34.680749 21786 leader_election.cc:302] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 187f342bdefd4b27a41fa3da3aaea0ad, b88194a8533f4d1e977904b2e7180d34, bc9b2d40233f4c9ab9545fb6184b0852; no voters:
I20260927 23:08:34.681016 22179 raft_consensus.cc:2781] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:34.681419 22179 raft_consensus.cc:686] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 1 LEADER]: Becoming Leader. State: Replica: bc9b2d40233f4c9ab9545fb6184b0852, State: Running, Role: LEADER
I20260927 23:08:34.681867 22179 consensus_queue.cc:237] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } }
I20260927 23:08:34.685777 21709 catalog_manager.cc:5495] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc9b2d40233f4c9ab9545fb6184b0852 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9b2d40233f4c9ab9545fb6184b0852" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 } health_report { overall_health: UNKNOWN } } }
I20260927 23:08:34.717654 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.712989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57226 (local address 127.3.240.193:44383)
0927 23:08:34.713194 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.713200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.713218 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:08:34.713381 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.713385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.713477 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.713599 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:08:34.713903 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.714547 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.715923 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.716278 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.716396 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.716745 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:08:34.717202 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:08:34.717292 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.717376 (+ 84us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.717455 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260927 23:08:34.724663 22110 raft_consensus.cc:1261] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 1 FOLLOWER]: Refusing update from remote peer bc9b2d40233f4c9ab9545fb6184b0852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:34.724721 21980 raft_consensus.cc:1261] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 1 FOLLOWER]: Refusing update from remote peer bc9b2d40233f4c9ab9545fb6184b0852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:34.725572 22179 consensus_queue.cc:1035] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "187f342bdefd4b27a41fa3da3aaea0ad" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:34.725832 22184 consensus_queue.cc:1035] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b88194a8533f4d1e977904b2e7180d34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260927 23:08:34.728060 22026 tablet.cc:2369] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:34.737771 22187 mvcc.cc:205] Tried to move back new op lower bound from 7334094908303659008 to 7334094908140003328. Current Snapshot: MvccSnapshot[applied={T|T < 7334094908303659008}]
I20260927 23:08:34.748149 21708 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:34.748414 21708 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:34.751844 21708 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dc25005c181346d6890222d9ec9aa163: Sending DeleteTablet for 3 replicas of tablet 7e09f90c3e9646669b2b53e894dd74e4
I20260927 23:08:34.752772 21830 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e09f90c3e9646669b2b53e894dd74e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:34 UTC) from {username='slave'} at 127.0.0.1:57224
I20260927 23:08:34.752856 21960 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e09f90c3e9646669b2b53e894dd74e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:34 UTC) from {username='slave'} at 127.0.0.1:35036
I20260927 23:08:34.753060 22090 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e09f90c3e9646669b2b53e894dd74e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:34 UTC) from {username='slave'} at 127.0.0.1:51314
I20260927 23:08:34.754843 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 21768
I20260927 23:08:34.765749 22198 tablet_replica.cc:330] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: stopping tablet replica
I20260927 23:08:34.766388 22198 raft_consensus.cc:2229] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:34.767053 22198 raft_consensus.cc:2258] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:34.768033 22197 tablet_replica.cc:330] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: stopping tablet replica
I20260927 23:08:34.769579 22198 ts_tablet_manager.cc:1905] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:34.770104 22197 raft_consensus.cc:2229] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:34.770578 22197 raft_consensus.cc:2258] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:34.772853 22197 ts_tablet_manager.cc:1905] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:34.776117 22196 tablet_replica.cc:330] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: stopping tablet replica
I20260927 23:08:34.776538 22196 raft_consensus.cc:2229] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:08:34.776623 22198 ts_tablet_manager.cc:1918] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:34.776821 22198 log.cc:1196] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/7e09f90c3e9646669b2b53e894dd74e4
I20260927 23:08:34.777356 22198 ts_tablet_manager.cc:1939] T 7e09f90c3e9646669b2b53e894dd74e4 P 187f342bdefd4b27a41fa3da3aaea0ad: Deleting consensus metadata
I20260927 23:08:34.778855 22196 raft_consensus.cc:2258] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:34.779510 21696 catalog_manager.cc:4841] TS 187f342bdefd4b27a41fa3da3aaea0ad (127.3.240.195:43931): tablet 7e09f90c3e9646669b2b53e894dd74e4 (table test-table [id=f81030ef1f264e28bb39db26549388a9]) successfully deleted
I20260927 23:08:34.780776 22197 ts_tablet_manager.cc:1918] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:34.780984 22197 log.cc:1196] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/7e09f90c3e9646669b2b53e894dd74e4
I20260927 23:08:34.781265 22169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.756316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57238 (local address 127.3.240.193:44383)
0927 23:08:34.756534 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.756544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.771429 (+ 14885us) server_negotiation.cc:407] Connection header received
0927 23:08:34.771701 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.771706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.771814 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.771891 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:34.772531 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.773806 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.776752 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.777425 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.777516 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.779496 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:34.779506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:34.779528 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:34.780318 (+ 790us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:34.780376 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:34.780416 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:34.780522 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.780749 (+ 227us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.780849 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260927 23:08:34.781781 22196 ts_tablet_manager.cc:1905] T 7e09f90c3e9646669b2b53e894dd74e4 P bc9b2d40233f4c9ab9545fb6184b0852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:34.782085 22197 ts_tablet_manager.cc:1939] T 7e09f90c3e9646669b2b53e894dd74e4 P b88194a8533f4d1e977904b2e7180d34: Deleting consensus metadata
I20260927 23:08:34.782316 21890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:34.783586 21695 catalog_manager.cc:4841] TS b88194a8533f4d1e977904b2e7180d34 (127.3.240.194:45405): tablet 7e09f90c3e9646669b2b53e894dd74e4 (table test-table [id=f81030ef1f264e28bb39db26549388a9]) successfully deleted
I20260927 23:08:34.874603 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 21768
I20260927 23:08:34.892393 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 21898
W20260927 23:08:34.892381 21694 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:44383 (error 108)
I20260927 23:08:34.894889 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.894434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44383 (local address 127.0.0.1:57242)
0927 23:08:34.894609 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.894723 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260927 23:08:34.895151 21694 catalog_manager.cc:4584] TS bc9b2d40233f4c9ab9545fb6184b0852 (127.3.240.193:44383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e09f90c3e9646669b2b53e894dd74e4: Network error: Client connection negotiation failed: client connection to 127.3.240.193:44383: connect: Connection refused (error 111)
I20260927 23:08:34.899441 22170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.893591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.3.240.194:45405)
0927 23:08:34.893794 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:34.893799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:34.893814 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:08:34.893867 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:34.893870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:34.893975 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:34.894089 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:34.894533 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.895468 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.896656 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:34.897023 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:34.897110 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:34.898155 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:34.898162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:34.898174 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:34.898685 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:34.898731 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:34.898749 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:34.898828 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:34.898976 (+ 148us) server_negotiation.cc:299] Negotiation successful
0927 23:08:34.899142 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 23:08:34.900082 22020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:34.948789 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.948334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44383 (local address 127.0.0.1:57252)
0927 23:08:34.948555 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.948638 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260927 23:08:34.949764 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:34.949451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44383 (local address 127.0.0.1:57258)
0927 23:08:34.949579 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:34.949651 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260927 23:08:34.988099 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 21898
I20260927 23:08:35.004348 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 22028
I20260927 23:08:35.010169 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:35.009787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44383 (local address 127.0.0.1:57274)
0927 23:08:35.009954 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:35.010028 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260927 23:08:35.011183 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:35.010812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44383 (local address 127.0.0.1:57286)
0927 23:08:35.010938 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:35.011058 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260927 23:08:35.014429 22172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:35.008676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51318 (local address 127.3.240.195:43931)
0927 23:08:35.008827 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:35.008832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:35.008849 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:08:35.008986 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:35.008991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:35.009072 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:35.009167 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:35.009617 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:35.010411 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:35.011823 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:35.012212 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:35.012307 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:35.013352 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:35.013359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:35.013373 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:35.013936 (+ 563us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:35.013976 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:35.013993 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:35.014077 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:35.014164 (+ 87us) server_negotiation.cc:299] Negotiation successful
0927 23:08:35.014238 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 23:08:35.014961 22150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:35.091679 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:35.091133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44383 (local address 127.0.0.1:57288)
0927 23:08:35.091360 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:35.091468 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260927 23:08:35.092729 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:35.092323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44383 (local address 127.0.0.1:57292)
0927 23:08:35.092519 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:35.092603 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:44383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260927 23:08:35.096110 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 22028
I20260927 23:08:35.115712 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 21678
I20260927 23:08:35.116866 21739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:35.223496 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 21678
2026-09-27T23:08:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2673 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:35 dist-test-slave-lmfp krb5kdc[22207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:35 dist-test-slave-lmfp krb5kdc[22207](info): set up 2 sockets
Sep 27 23:08:35 dist-test-slave-lmfp krb5kdc[22207](info): commencing operation
krb5kdc: starting...
W20260927 23:08:36.294930 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:36 dist-test-slave-lmfp krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:08:36.522583 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44099
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44099
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:36.846052 22220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:36.846334 22220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:36.846465 22220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:36.853652 22220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:36.853740 22220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:36.853804 22220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:36.853904 22220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:36.853988 22220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:36.863667 22220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44099
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:36.864892 22220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:36.866230 22220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:36.878073 22225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:36.878175 22226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:36.878122 22228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:36.879122 22220 server_base.cc:1034] running on GCE node
I20260927 23:08:36.879820 22220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:36.881086 22220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:36.882354 22220 hybrid_clock.cc:648] HybridClock initialized: now 1790550516882294 us; error 49 us; skew 500 ppm
I20260927 23:08:36.882709 22220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:36 dist-test-slave-lmfp krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550516, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:36.885727 22220 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:08:36.887180 22220 webserver.cc:460] Webserver started at http://127.3.240.254:38119/ using document root <none> and password file <none>
I20260927 23:08:36.887853 22220 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:36.888008 22220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:36.888319 22220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:36.891337 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "2f7a3c217c29419dae6c689d94c45ede"
format_stamp: "Formatted at 2026-09-27 23:08:36 on dist-test-slave-lmfp"
server_key: "afde98b1db4316777346f914552920ba"
server_key_iv: "05c2cfb5b72b861f95e50ce1fa4a3760"
server_key_version: "encryptionkey@0"
I20260927 23:08:36.892112 22220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2f7a3c217c29419dae6c689d94c45ede"
format_stamp: "Formatted at 2026-09-27 23:08:36 on dist-test-slave-lmfp"
server_key: "afde98b1db4316777346f914552920ba"
server_key_iv: "05c2cfb5b72b861f95e50ce1fa4a3760"
server_key_version: "encryptionkey@0"
I20260927 23:08:36.897236 22220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:08:36.900529 22235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:36.901813 22220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 23:08:36.901991 22220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "2f7a3c217c29419dae6c689d94c45ede"
format_stamp: "Formatted at 2026-09-27 23:08:36 on dist-test-slave-lmfp"
server_key: "afde98b1db4316777346f914552920ba"
server_key_iv: "05c2cfb5b72b861f95e50ce1fa4a3760"
server_key_version: "encryptionkey@0"
I20260927 23:08:36.902189 22220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:36.917963 22220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:36.921696 22220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:36.921984 22220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:36.939329 22220 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:44099
I20260927 23:08:36.939358 22286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:44099 every 8 connection(s)
I20260927 23:08:36.940595 22220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:36.944049 22287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:36.944603 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 22220
I20260927 23:08:36.944787 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:36.945204 4035 external_mini_cluster.cc:1428] Setting key 85f4b29bf1693c5d596cd33e7f030a90
I20260927 23:08:36.957053 22287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede: Bootstrap starting.
Sep 27 23:08:36 dist-test-slave-lmfp krb5kdc[22207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:36.960662 22287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:36.962044 22287 log.cc:824] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:36.965265 22287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede: No bootstrap required, opened a new log
I20260927 23:08:36.968657 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:36.946864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.3.240.254:44099)
0927 23:08:36.947577 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:36.947587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:36.947641 (+ 54us) server_negotiation.cc:407] Connection header received
0927 23:08:36.949628 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:36.949649 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:36.950000 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:36.950326 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:36.952067 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:36.953305 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:36.955128 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:36.955750 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:36.955870 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:36.960487 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:36.960498 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:36.960517 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:36.960568 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:36.964353 (+ 3785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:36.964996 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:36.965004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:36.965011 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:36.965190 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:36.965565 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:36.965569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:36.965572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:36.966304 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:36.966633 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:36.966899 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 23:08:36.967343 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":352,"threads_started":1}
I20260927 23:08:36.974015 22287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f7a3c217c29419dae6c689d94c45ede" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44099 } }
I20260927 23:08:36.974320 22287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:36.974387 22287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7a3c217c29419dae6c689d94c45ede, State: Initialized, Role: FOLLOWER
I20260927 23:08:36.974819 22287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f7a3c217c29419dae6c689d94c45ede" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44099 } }
I20260927 23:08:36.975075 22287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:36.975220 22287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:36.975407 22287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:36.977419 22287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f7a3c217c29419dae6c689d94c45ede" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44099 } }
I20260927 23:08:36.977876 22287 leader_election.cc:302] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f7a3c217c29419dae6c689d94c45ede; no voters:
I20260927 23:08:36.978570 22287 leader_election.cc:288] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:36.978719 22292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:36.979449 22292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [term 1 LEADER]: Becoming Leader. State: Replica: 2f7a3c217c29419dae6c689d94c45ede, State: Running, Role: LEADER
I20260927 23:08:36.980007 22292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f7a3c217c29419dae6c689d94c45ede" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44099 } }
I20260927 23:08:36.980614 22287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:36.982579 22293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f7a3c217c29419dae6c689d94c45ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f7a3c217c29419dae6c689d94c45ede" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44099 } } }
I20260927 23:08:36.982909 22293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:36.983373 22294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f7a3c217c29419dae6c689d94c45ede. Latest consensus state: current_term: 1 leader_uuid: "2f7a3c217c29419dae6c689d94c45ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f7a3c217c29419dae6c689d94c45ede" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44099 } } }
I20260927 23:08:36.983664 22294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:36.985219 22298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:36.989828 22298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:36.998119 22298 catalog_manager.cc:1347] Generated new cluster ID: 01b5a35d3ecf41a2bb5bb99dbb37d15c
I20260927 23:08:36.998229 22298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:37.036859 22298 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:37.038165 22298 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:37.044806 22298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede: Generated new TSK 0
I20260927 23:08:37.045614 22298 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:08:37.117677 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44099
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:37.446118 22313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:37.446394 22313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:37.446487 22313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:37.453794 22313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:37.453922 22313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:37.454049 22313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:08:37.463423 22313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:37.464634 22313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:37.465905 22313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:37.478660 22321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:37.478710 22318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:37.478739 22319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:37.479425 22313 server_base.cc:1034] running on GCE node
I20260927 23:08:37.480057 22313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:37.480986 22313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:37.482189 22313 hybrid_clock.cc:648] HybridClock initialized: now 1790550517482156 us; error 40 us; skew 500 ppm
I20260927 23:08:37.482534 22313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:37 dist-test-slave-lmfp krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:37.485589 22313 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:08:37.486964 22313 webserver.cc:460] Webserver started at http://127.3.240.193:44309/ using document root <none> and password file <none>
I20260927 23:08:37.487741 22313 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:37.487895 22313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:37.488190 22313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:37.490904 22313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "d0dea7e9bd204f61a79b9fbebd240113"
format_stamp: "Formatted at 2026-09-27 23:08:37 on dist-test-slave-lmfp"
server_key: "7a7774f052931a458a6002a41d7aab52"
server_key_iv: "f16416759af6c57973e1bf596243a617"
server_key_version: "encryptionkey@0"
I20260927 23:08:37.491750 22313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "d0dea7e9bd204f61a79b9fbebd240113"
format_stamp: "Formatted at 2026-09-27 23:08:37 on dist-test-slave-lmfp"
server_key: "7a7774f052931a458a6002a41d7aab52"
server_key_iv: "f16416759af6c57973e1bf596243a617"
server_key_version: "encryptionkey@0"
I20260927 23:08:37.496651 22313 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:08:37.499984 22328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:37.501263 22313 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 23:08:37.501430 22313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "d0dea7e9bd204f61a79b9fbebd240113"
format_stamp: "Formatted at 2026-09-27 23:08:37 on dist-test-slave-lmfp"
server_key: "7a7774f052931a458a6002a41d7aab52"
server_key_iv: "f16416759af6c57973e1bf596243a617"
server_key_version: "encryptionkey@0"
I20260927 23:08:37.501633 22313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:37.543151 22313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:37.547322 22313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:37.547608 22313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:37.548619 22313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:37.549847 22313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:37.549940 22313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:37.550081 22313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:37.550163 22313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:37.583109 22313 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:33039
I20260927 23:08:37.583160 22440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:33039 every 8 connection(s)
I20260927 23:08:37.584378 22313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:08:37.585397 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 22313
I20260927 23:08:37.585579 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:08:37.586024 4035 external_mini_cluster.cc:1428] Setting key 505d5eda78b9306fa04a288e37508178
Sep 27 23:08:37 dist-test-slave-lmfp krb5kdc[22207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550517, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:37.605990 22444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:37.587689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44099 (local address 127.3.240.193:51685)
0927 23:08:37.588514 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:37.588571 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:37.590229 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:37.590907 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:37.590915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:37.591449 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:37.591988 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:37.592005 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:37.592999 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:37.593003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:37.594293 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:37.594304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:37.595149 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:37.595159 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:37.595404 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:37.596071 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:37.596091 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:37.598166 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:37.601196 (+ 3030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:37.601205 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:37.601220 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:37.601636 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:37.601998 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:37.602003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:37.602007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:37.602278 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:37.603084 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:37.603092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:37.603512 (+ 420us) client_negotiation.cc:769] Sending connection context
0927 23:08:37.603834 (+ 322us) client_negotiation.cc:240] Negotiation successful
0927 23:08:37.604405 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":372,"threads_started":1}
I20260927 23:08:37.606801 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:37.587069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:51685 (local address 127.3.240.254:44099)
0927 23:08:37.587564 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:37.587569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:37.588688 (+ 1119us) server_negotiation.cc:407] Connection header received
0927 23:08:37.590479 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:37.590485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:37.590564 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:37.590685 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:37.592146 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:37.592838 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:37.594433 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:37.594888 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:37.595059 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:37.598367 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:37.598371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:37.598375 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:37.598406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:37.601045 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:37.601761 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:37.601771 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:37.601773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:37.601888 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:37.602442 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:37.602450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:37.602456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:37.602879 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:37.603067 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:37.606339 (+ 3272us) server_negotiation.cc:299] Negotiation successful
0927 23:08:37.606577 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":234,"threads_started":1}
I20260927 23:08:37.608549 22441 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44099
I20260927 23:08:37.608862 22441 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260927 23:08:37.609620 22441 heartbeater.cc:511] Master 127.3.240.254:44099 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260927 23:08:37.611979 22252 ts_manager.cc:194] Registered new tserver with Master: d0dea7e9bd204f61a79b9fbebd240113 (127.3.240.193:33039)
I20260927 23:08:37.614143 22252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:51685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:08:37.639685 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44099
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:37.970070 22447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:37.970309 22447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:37.970395 22447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:37.977015 22447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:37.977098 22447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:37.977195 22447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:08:37.986323 22447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:37.987504 22447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:37.988646 22447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:38.000916 22452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:38.000946 22453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:38.001080 22455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:38.001714 22447 server_base.cc:1034] running on GCE node
I20260927 23:08:38.002385 22447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:38.003430 22447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:38.004685 22447 hybrid_clock.cc:648] HybridClock initialized: now 1790550518004617 us; error 52 us; skew 500 ppm
I20260927 23:08:38.005049 22447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:38 dist-test-slave-lmfp krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:38.012746 22447 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:08:38.014067 22447 webserver.cc:460] Webserver started at http://127.3.240.194:43747/ using document root <none> and password file <none>
I20260927 23:08:38.014711 22447 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:38.014847 22447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:38.015200 22447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:38.017745 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "1d70b91e37c64f6fbacfa5943fd2e45e"
format_stamp: "Formatted at 2026-09-27 23:08:38 on dist-test-slave-lmfp"
server_key: "0ff3656a95461c593fb321ba2f5f2852"
server_key_iv: "53b1c2ca88e0f893d9b6711a732b73e5"
server_key_version: "encryptionkey@0"
I20260927 23:08:38.018514 22447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "1d70b91e37c64f6fbacfa5943fd2e45e"
format_stamp: "Formatted at 2026-09-27 23:08:38 on dist-test-slave-lmfp"
server_key: "0ff3656a95461c593fb321ba2f5f2852"
server_key_iv: "53b1c2ca88e0f893d9b6711a732b73e5"
server_key_version: "encryptionkey@0"
I20260927 23:08:38.023351 22447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.005s
I20260927 23:08:38.026523 22462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:38.027669 22447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:08:38.027829 22447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "1d70b91e37c64f6fbacfa5943fd2e45e"
format_stamp: "Formatted at 2026-09-27 23:08:38 on dist-test-slave-lmfp"
server_key: "0ff3656a95461c593fb321ba2f5f2852"
server_key_iv: "53b1c2ca88e0f893d9b6711a732b73e5"
server_key_version: "encryptionkey@0"
I20260927 23:08:38.028008 22447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:38.043385 22447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:38.047380 22447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:38.047662 22447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:38.048484 22447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:38.049671 22447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:38.049791 22447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:38.049963 22447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:38.050035 22447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:38.082885 22447 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:41109
I20260927 23:08:38.082927 22574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:41109 every 8 connection(s)
I20260927 23:08:38.084131 22447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:08:38.085259 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 22447
I20260927 23:08:38.085424 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:08:38.085999 4035 external_mini_cluster.cc:1428] Setting key 25d94f40bf6c367315990b9005750278
Sep 27 23:08:38 dist-test-slave-lmfp krb5kdc[22207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:38.107250 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.086802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:39141 (local address 127.3.240.254:44099)
0927 23:08:38.087054 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.087059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.088412 (+ 1353us) server_negotiation.cc:407] Connection header received
0927 23:08:38.091169 (+ 2757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.091173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.091232 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.091307 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:38.093269 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.093936 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.096404 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.096819 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.096909 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.100284 (+ 3375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:38.100293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:38.100301 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:38.100345 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:38.103921 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:38.104603 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:38.104611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:38.104618 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:38.104773 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:38.105271 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:38.105279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:38.105285 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:38.105719 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:38.105876 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.106674 (+ 798us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.106892 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:08:38.108173 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.087445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44099 (local address 127.3.240.194:39141)
0927 23:08:38.088259 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:38.088315 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:38.090878 (+ 2563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:38.091524 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:38.091536 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:38.092311 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:38.093099 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.093126 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.094096 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.094104 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.096270 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.096287 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.097057 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.097066 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.097279 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.097918 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:38.097941 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:38.100063 (+ 2122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:38.104057 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:38.104064 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:38.104075 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:38.104480 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:38.104880 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:38.104883 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:38.104886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:38.105107 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:38.105851 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:38.105861 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:38.106206 (+ 345us) client_negotiation.cc:769] Sending connection context
0927 23:08:38.106574 (+ 368us) client_negotiation.cc:240] Negotiation successful
0927 23:08:38.107080 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":307,"threads_started":1}
I20260927 23:08:38.109584 22575 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44099
I20260927 23:08:38.109866 22575 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:38.110515 22575 heartbeater.cc:511] Master 127.3.240.254:44099 requested a full tablet report, sending...
I20260927 23:08:38.111835 22252 ts_manager.cc:194] Registered new tserver with Master: 1d70b91e37c64f6fbacfa5943fd2e45e (127.3.240.194:41109)
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260927 23:08:38.112650 22252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:39141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:08:38.140210 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44099
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:38.447510 22580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:38.447793 22580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:38.447937 22580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:38.454689 22580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:38.454818 22580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:38.455010 22580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:08:38.464182 22580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:38.465474 22580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:38.466737 22580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:38.479386 22585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:38.479414 22586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:38.479259 22588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:38.480039 22580 server_base.cc:1034] running on GCE node
I20260927 23:08:38.480558 22580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:38.481436 22580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:38.482659 22580 hybrid_clock.cc:648] HybridClock initialized: now 1790550518482584 us; error 65 us; skew 500 ppm
I20260927 23:08:38.483028 22580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:38 dist-test-slave-lmfp krb5kdc[22207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:38.486102 22580 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:08:38.487680 22580 webserver.cc:460] Webserver started at http://127.3.240.195:38105/ using document root <none> and password file <none>
I20260927 23:08:38.488416 22580 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:38.488523 22580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:38.488817 22580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:38.491636 22580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "e1db5d3e54cd4e3d974e5b70362e2a87"
format_stamp: "Formatted at 2026-09-27 23:08:38 on dist-test-slave-lmfp"
server_key: "3cf191ae6ed92139e8d5e3aeb7179ea3"
server_key_iv: "ee38fe3ffc043941c8d21a863248f92e"
server_key_version: "encryptionkey@0"
I20260927 23:08:38.492339 22580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "e1db5d3e54cd4e3d974e5b70362e2a87"
format_stamp: "Formatted at 2026-09-27 23:08:38 on dist-test-slave-lmfp"
server_key: "3cf191ae6ed92139e8d5e3aeb7179ea3"
server_key_iv: "ee38fe3ffc043941c8d21a863248f92e"
server_key_version: "encryptionkey@0"
I20260927 23:08:38.497057 22580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:08:38.500185 22595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:38.501411 22580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:08:38.501601 22580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "e1db5d3e54cd4e3d974e5b70362e2a87"
format_stamp: "Formatted at 2026-09-27 23:08:38 on dist-test-slave-lmfp"
server_key: "3cf191ae6ed92139e8d5e3aeb7179ea3"
server_key_iv: "ee38fe3ffc043941c8d21a863248f92e"
server_key_version: "encryptionkey@0"
I20260927 23:08:38.501796 22580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:38.531247 22580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:38.535265 22580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:38.535554 22580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:38.536471 22580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:38.537781 22580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:38.537884 22580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:38.538034 22580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:38.538115 22580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:38.570813 22580 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:36035
I20260927 23:08:38.570870 22707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:36035 every 8 connection(s)
I20260927 23:08:38.572084 22580 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:08:38.580757 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 22580
I20260927 23:08:38.580931 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:08:38.581485 4035 external_mini_cluster.cc:1428] Setting key 16dbbb8444f30b13c2ffc9849d3db489
Sep 27 23:08:38 dist-test-slave-lmfp krb5kdc[22207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550518, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:38.592006 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.574533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:37931 (local address 127.3.240.254:44099)
0927 23:08:38.574700 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.574704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.575843 (+ 1139us) server_negotiation.cc:407] Connection header received
0927 23:08:38.577527 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.577532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.577639 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.577744 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:38.579105 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.579814 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.581797 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.582429 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.582567 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.586426 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:38.586433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:38.586442 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:38.586473 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:38.589135 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:38.589802 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:38.589807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:38.589809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:38.589931 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:38.590352 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:38.590357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:38.590361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:38.590615 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:38.590764 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.591472 (+ 708us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.591677 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 23:08:38.592806 22710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.575050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44099 (local address 127.3.240.195:37931)
0927 23:08:38.575696 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:38.575734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:38.577282 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:38.577948 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:38.577955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:38.578422 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:38.578926 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.578941 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.579962 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.579966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.581638 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.581652 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.582723 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.582730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.583024 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.583767 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:38.583795 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:38.586224 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:38.589264 (+ 3040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:38.589273 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:38.589286 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:38.589677 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:38.590057 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:38.590062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:38.590066 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:38.590238 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:38.590735 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:38.590742 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:38.591134 (+ 392us) client_negotiation.cc:769] Sending connection context
0927 23:08:38.591369 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 23:08:38.591829 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":246,"threads_started":1}
I20260927 23:08:38.594003 22708 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44099
I20260927 23:08:38.594311 22708 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:38.595201 22708 heartbeater.cc:511] Master 127.3.240.254:44099 requested a full tablet report, sending...
I20260927 23:08:38.596642 22252 ts_manager.cc:194] Registered new tserver with Master: e1db5d3e54cd4e3d974e5b70362e2a87 (127.3.240.195:36035)
I20260927 23:08:38.597383 22252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:37931
I20260927 23:08:38.609742 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:08:38.617250 22441 heartbeater.cc:503] Master 127.3.240.254:44099 was elected leader, sending a full tablet report...
I20260927 23:08:38.625669 22443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.613710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.3.240.254:44099)
0927 23:08:38.613883 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.613887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.614252 (+ 365us) server_negotiation.cc:407] Connection header received
0927 23:08:38.614374 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.614377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.614430 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.614508 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:38.615859 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.616537 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.617898 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.618317 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.618405 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.620962 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:38.620967 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:38.620970 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:38.621007 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:38.623515 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:38.624149 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:38.624152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:38.624154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:38.624266 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:38.624600 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:38.624602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:38.624604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:38.624872 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:38.625056 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.625364 (+ 308us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.625475 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 23:08:38.629197 22252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:08:38.635910 22252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:08:38.673347 22720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.646946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:41109 (local address 127.0.0.1:51174)
0927 23:08:38.662272 (+ 15326us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:38.662330 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:38.662551 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:38.666311 (+ 3760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:38.666315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:38.666341 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:38.667093 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.667101 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.668480 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.668484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.670236 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.670247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.672283 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.672290 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.672475 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.672542 (+ 67us) client_negotiation.cc:769] Sending connection context
0927 23:08:38.672681 (+ 139us) client_negotiation.cc:240] Negotiation successful
0927 23:08:38.672861 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14987,"mutex_wait_us":54,"thread_start_us":15819,"threads_started":1}
I20260927 23:08:38.673661 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.648002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46780 (local address 127.3.240.195:36035)
0927 23:08:38.664994 (+ 16992us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.665004 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.665700 (+ 696us) server_negotiation.cc:407] Connection header received
0927 23:08:38.665977 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.665985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.666162 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.666311 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:38.667295 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.668609 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.671104 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.672215 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.672331 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.672550 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.673092 (+ 542us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.673272 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16776,"thread_start_us":16629,"threads_started":1}
I20260927 23:08:38.673538 22723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.647495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:36035 (local address 127.0.0.1:46780)
0927 23:08:38.665428 (+ 17933us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:38.665446 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:38.665578 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:38.666565 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:38.666570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:38.666599 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:38.667085 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.667098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.668742 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.668749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.670933 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.670942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.672697 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.672703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.672868 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.672890 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 23:08:38.673019 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 23:08:38.673137 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17804,"mutex_wait_us":31,"thread_start_us":17675,"threads_started":1}
I20260927 23:08:38.674275 22721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.647061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33039 (local address 127.0.0.1:53798)
0927 23:08:38.664592 (+ 17531us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:38.664613 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:38.664891 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:38.665770 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:38.665777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:38.665806 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:38.666236 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.666248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.668159 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.668164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.670585 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.670599 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.672643 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.672651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.672816 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.672844 (+ 28us) client_negotiation.cc:769] Sending connection context
0927 23:08:38.673799 (+ 955us) client_negotiation.cc:240] Negotiation successful
0927 23:08:38.673935 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17369,"mutex_wait_us":51,"thread_start_us":17209,"threads_started":1}
I20260927 23:08:38.674634 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.647145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.3.240.194:41109)
0927 23:08:38.665756 (+ 18611us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.665764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.665786 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:08:38.665877 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.665883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.666093 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.666702 (+ 609us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:38.667311 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.668347 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.670379 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.671813 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.671944 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.672136 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.674152 (+ 2016us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.674345 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18466,"thread_start_us":18338,"threads_started":1}
I20260927 23:08:38.674976 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.647651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.3.240.193:33039)
0927 23:08:38.664104 (+ 16453us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.664117 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.664757 (+ 640us) server_negotiation.cc:407] Connection header received
0927 23:08:38.665092 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.665103 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.665405 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.665620 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:38.666423 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.668029 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.670744 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.672160 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.672289 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.672497 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.674467 (+ 1970us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.674610 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16219,"thread_start_us":16033,"threads_started":1}
I20260927 23:08:38.677986 22376 tablet_service.cc:1467] Processing CreateTablet for tablet 2cab533981e64446b93b2857795ea8ce (DEFAULT_TABLE table=test-table [id=3f6c62153a7b4cb1922e4a3435adcbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:38.677961 22510 tablet_service.cc:1467] Processing CreateTablet for tablet 2cab533981e64446b93b2857795ea8ce (DEFAULT_TABLE table=test-table [id=3f6c62153a7b4cb1922e4a3435adcbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:38.677963 22643 tablet_service.cc:1467] Processing CreateTablet for tablet 2cab533981e64446b93b2857795ea8ce (DEFAULT_TABLE table=test-table [id=3f6c62153a7b4cb1922e4a3435adcbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:38.679330 22510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cab533981e64446b93b2857795ea8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:38.679330 22643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cab533981e64446b93b2857795ea8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:38.679337 22376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cab533981e64446b93b2857795ea8ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:38.689478 22726 tablet_bootstrap.cc:492] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Bootstrap starting.
I20260927 23:08:38.692662 22727 tablet_bootstrap.cc:492] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Bootstrap starting.
I20260927 23:08:38.693300 22726 tablet_bootstrap.cc:654] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:38.693475 22728 tablet_bootstrap.cc:492] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Bootstrap starting.
I20260927 23:08:38.694594 22726 log.cc:824] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:38.696768 22728 tablet_bootstrap.cc:654] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:38.696789 22727 tablet_bootstrap.cc:654] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:38.697659 22727 log.cc:824] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:38.697715 22726 tablet_bootstrap.cc:492] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: No bootstrap required, opened a new log
I20260927 23:08:38.697914 22728 log.cc:824] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:38.697992 22726 ts_tablet_manager.cc:1397] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260927 23:08:38.705468 22727 tablet_bootstrap.cc:492] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: No bootstrap required, opened a new log
I20260927 23:08:38.705597 22728 tablet_bootstrap.cc:492] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: No bootstrap required, opened a new log
I20260927 23:08:38.705706 22727 ts_tablet_manager.cc:1397] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260927 23:08:38.705880 22728 ts_tablet_manager.cc:1397] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20260927 23:08:38.710209 22726 raft_consensus.cc:348] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
I20260927 23:08:38.710604 22726 raft_consensus.cc:374] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:38.710692 22726 raft_consensus.cc:729] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0dea7e9bd204f61a79b9fbebd240113, State: Initialized, Role: FOLLOWER
I20260927 23:08:38.711263 22726 consensus_queue.cc:260] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
I20260927 23:08:38.713330 22728 raft_consensus.cc:348] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
I20260927 23:08:38.713625 22728 raft_consensus.cc:374] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:38.713716 22728 raft_consensus.cc:729] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1db5d3e54cd4e3d974e5b70362e2a87, State: Initialized, Role: FOLLOWER
I20260927 23:08:38.714246 22728 consensus_queue.cc:260] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
I20260927 23:08:38.715229 22708 heartbeater.cc:503] Master 127.3.240.254:44099 was elected leader, sending a full tablet report...
I20260927 23:08:38.715765 22728 ts_tablet_manager.cc:1428] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260927 23:08:38.716542 22727 raft_consensus.cc:348] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
I20260927 23:08:38.716917 22727 raft_consensus.cc:374] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:38.717020 22727 raft_consensus.cc:729] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d70b91e37c64f6fbacfa5943fd2e45e, State: Initialized, Role: FOLLOWER
I20260927 23:08:38.717536 22727 consensus_queue.cc:260] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
I20260927 23:08:38.718458 22726 ts_tablet_manager.cc:1428] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Time spent starting tablet: real 0.020s user 0.008s sys 0.009s
I20260927 23:08:38.719795 22575 heartbeater.cc:503] Master 127.3.240.254:44099 was elected leader, sending a full tablet report...
I20260927 23:08:38.720335 22727 ts_tablet_manager.cc:1428] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260927 23:08:38.802114 22734 raft_consensus.cc:482] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:08:38.802412 22734 raft_consensus.cc:504] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
I20260927 23:08:38.804113 22734 leader_election.cc:288] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0dea7e9bd204f61a79b9fbebd240113 (127.3.240.193:33039), e1db5d3e54cd4e3d974e5b70362e2a87 (127.3.240.195:36035)
I20260927 23:08:38.812089 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.804531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42215 (local address 127.3.240.193:33039)
0927 23:08:38.805072 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.805081 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.805171 (+ 90us) server_negotiation.cc:407] Connection header received
0927 23:08:38.805483 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.805487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.805625 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.805742 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:38.806294 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.807406 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.810013 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.811218 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.811367 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.811436 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.811723 (+ 287us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.811850 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260927 23:08:38.812321 22735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.804274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33039 (local address 127.3.240.194:42215)
0927 23:08:38.805067 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:38.805086 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:38.805298 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:38.805762 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:38.805766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:38.805790 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:38.806147 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.806164 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.807551 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.807554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.809851 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.809865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.811437 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.811442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.811558 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.811585 (+ 27us) client_negotiation.cc:769] Sending connection context
0927 23:08:38.811680 (+ 95us) client_negotiation.cc:240] Negotiation successful
0927 23:08:38.811798 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":117,"thread_start_us":424,"threads_started":1}
I20260927 23:08:38.812448 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.804875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:40165 (local address 127.3.240.195:36035)
0927 23:08:38.805208 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.805217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.805646 (+ 429us) server_negotiation.cc:407] Connection header received
0927 23:08:38.805902 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.805906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.805962 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.806062 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:38.806637 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.807668 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.810013 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.811194 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.811367 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.811436 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.811948 (+ 512us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.812070 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260927 23:08:38.812511 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.804497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:36035 (local address 127.3.240.194:40165)
0927 23:08:38.805533 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:38.805546 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:38.805752 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:38.806088 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:38.806094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:38.806122 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:38.806505 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.806512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.807776 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.807782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.809851 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:38.809865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.811586 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.811590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:38.811782 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.811816 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 23:08:38.811904 (+ 88us) client_negotiation.cc:240] Negotiation successful
0927 23:08:38.812031 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":43,"thread_start_us":718,"threads_started":1}
I20260927 23:08:38.813416 22396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cab533981e64446b93b2857795ea8ce" candidate_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dea7e9bd204f61a79b9fbebd240113" is_pre_election: true
I20260927 23:08:38.813807 22663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cab533981e64446b93b2857795ea8ce" candidate_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" is_pre_election: true
I20260927 23:08:38.814034 22396 raft_consensus.cc:2454] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d70b91e37c64f6fbacfa5943fd2e45e in term 0.
I20260927 23:08:38.814347 22663 raft_consensus.cc:2454] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d70b91e37c64f6fbacfa5943fd2e45e in term 0.
I20260927 23:08:38.814738 22464 leader_election.cc:302] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d70b91e37c64f6fbacfa5943fd2e45e, d0dea7e9bd204f61a79b9fbebd240113; no voters:
I20260927 23:08:38.815192 22734 raft_consensus.cc:2781] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:08:38.815322 22734 raft_consensus.cc:482] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:08:38.815469 22734 raft_consensus.cc:3037] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:38.817497 22734 raft_consensus.cc:504] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
I20260927 23:08:38.818106 22734 leader_election.cc:288] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [CANDIDATE]: Term 1 election: Requested vote from peers d0dea7e9bd204f61a79b9fbebd240113 (127.3.240.193:33039), e1db5d3e54cd4e3d974e5b70362e2a87 (127.3.240.195:36035)
I20260927 23:08:38.818635 22663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cab533981e64446b93b2857795ea8ce" candidate_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87"
I20260927 23:08:38.818671 22396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cab533981e64446b93b2857795ea8ce" candidate_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0dea7e9bd204f61a79b9fbebd240113"
I20260927 23:08:38.818871 22663 raft_consensus.cc:3037] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:38.818868 22396 raft_consensus.cc:3037] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:38.821012 22396 raft_consensus.cc:2454] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d70b91e37c64f6fbacfa5943fd2e45e in term 1.
I20260927 23:08:38.821051 22663 raft_consensus.cc:2454] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d70b91e37c64f6fbacfa5943fd2e45e in term 1.
I20260927 23:08:38.821355 22464 leader_election.cc:302] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d70b91e37c64f6fbacfa5943fd2e45e, d0dea7e9bd204f61a79b9fbebd240113; no voters:
I20260927 23:08:38.821607 22734 raft_consensus.cc:2781] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:38.822041 22734 raft_consensus.cc:686] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 1 LEADER]: Becoming Leader. State: Replica: 1d70b91e37c64f6fbacfa5943fd2e45e, State: Running, Role: LEADER
I20260927 23:08:38.822475 22734 consensus_queue.cc:237] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } }
W20260927 23:08:38.823637 22709 tablet.cc:2369] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:38.826849 22251 catalog_manager.cc:5495] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d70b91e37c64f6fbacfa5943fd2e45e (127.3.240.194). New cstate: current_term: 1 leader_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d70b91e37c64f6fbacfa5943fd2e45e" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 } health_report { overall_health: UNKNOWN } } }
W20260927 23:08:38.835927 22576 tablet.cc:2369] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:08:38.836616 22442 tablet.cc:2369] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:38.848212 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.843669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.3.240.194:41109)
0927 23:08:38.843822 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.843826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.843961 (+ 135us) server_negotiation.cc:407] Connection header received
0927 23:08:38.844104 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.844107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.844179 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.844283 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:08:38.844572 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.845258 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.846474 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.846834 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.846914 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.847388 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:08:38.847791 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:08:38.847904 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.847996 (+ 92us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.848027 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 23:08:38.854943 22663 raft_consensus.cc:1261] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 1 FOLLOWER]: Refusing update from remote peer 1d70b91e37c64f6fbacfa5943fd2e45e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:38.854951 22396 raft_consensus.cc:1261] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 1 FOLLOWER]: Refusing update from remote peer 1d70b91e37c64f6fbacfa5943fd2e45e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:38.855827 22737 consensus_queue.cc:1035] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0dea7e9bd204f61a79b9fbebd240113" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:38.856119 22734 consensus_queue.cc:1035] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1db5d3e54cd4e3d974e5b70362e2a87" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:38.865876 22740 mvcc.cc:205] Tried to move back new op lower bound from 7334094925221634048 to 7334094925100748800. Current Snapshot: MvccSnapshot[applied={T|T < 7334094925221634048}]
I20260927 23:08:38.877826 22252 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:38.878060 22252 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:38.881269 22252 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f7a3c217c29419dae6c689d94c45ede: Sending DeleteTablet for 3 replicas of tablet 2cab533981e64446b93b2857795ea8ce
I20260927 23:08:38.882316 22376 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cab533981e64446b93b2857795ea8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:38 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53798
I20260927 23:08:38.882316 22643 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cab533981e64446b93b2857795ea8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:38 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:46780
I20260927 23:08:38.883322 22510 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cab533981e64446b93b2857795ea8ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:38 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:51174
I20260927 23:08:38.886693 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 22313
I20260927 23:08:38.900102 22751 tablet_replica.cc:330] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: stopping tablet replica
I20260927 23:08:38.900749 22751 raft_consensus.cc:2229] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:38.901450 22751 raft_consensus.cc:2258] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:38.903597 22751 ts_tablet_manager.cc:1905] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:38.905791 22750 tablet_replica.cc:330] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: stopping tablet replica
I20260927 23:08:38.907980 22750 raft_consensus.cc:2229] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:38.908489 22750 raft_consensus.cc:2258] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:38.911489 22752 tablet_replica.cc:330] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: stopping tablet replica
I20260927 23:08:38.911768 22751 ts_tablet_manager.cc:1918] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:38.911947 22751 log.cc:1196] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/2cab533981e64446b93b2857795ea8ce
I20260927 23:08:38.911917 22752 raft_consensus.cc:2229] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:08:38.912459 22751 ts_tablet_manager.cc:1939] T 2cab533981e64446b93b2857795ea8ce P d0dea7e9bd204f61a79b9fbebd240113: Deleting consensus metadata
I20260927 23:08:38.912590 22752 raft_consensus.cc:2258] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 27 23:08:38 dist-test-slave-lmfp krb5kdc[22207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:08:38.914870 22752 ts_tablet_manager.cc:1905] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:38.915437 22750 ts_tablet_manager.cc:1905] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:38.922506 22750 ts_tablet_manager.cc:1918] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:38.922746 22752 ts_tablet_manager.cc:1918] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:38.923187 22752 log.cc:1196] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/2cab533981e64446b93b2857795ea8ce
I20260927 23:08:38.923734 22724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:38.890791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.3.240.193:33039)
0927 23:08:38.891521 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:38.891528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:38.891547 (+ 19us) server_negotiation.cc:407] Connection header received
0927 23:08:38.891772 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:38.891780 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:38.891870 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:38.891967 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:38.894273 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.895755 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.906012 (+ 10257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:38.906721 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:38.906813 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:38.916176 (+ 9363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:38.916189 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:38.916205 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:38.916242 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:38.919697 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:38.920559 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:38.920567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:38.920570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:38.920750 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:38.921209 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:38.921214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:38.921217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:38.921644 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:38.921921 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:38.923093 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0927 23:08:38.923340 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614}
I20260927 23:08:38.923795 22237 catalog_manager.cc:4841] TS d0dea7e9bd204f61a79b9fbebd240113 (127.3.240.193:33039): tablet 2cab533981e64446b93b2857795ea8ce (table test-table [id=3f6c62153a7b4cb1922e4a3435adcbe3]) successfully deleted
I20260927 23:08:38.923781 22752 ts_tablet_manager.cc:1939] T 2cab533981e64446b93b2857795ea8ce P 1d70b91e37c64f6fbacfa5943fd2e45e: Deleting consensus metadata
I20260927 23:08:38.924271 22750 log.cc:1196] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/2cab533981e64446b93b2857795ea8ce
I20260927 23:08:38.924485 22436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:38.924831 22750 ts_tablet_manager.cc:1939] T 2cab533981e64446b93b2857795ea8ce P e1db5d3e54cd4e3d974e5b70362e2a87: Deleting consensus metadata
I20260927 23:08:38.925838 22239 catalog_manager.cc:4841] TS 1d70b91e37c64f6fbacfa5943fd2e45e (127.3.240.194:41109): tablet 2cab533981e64446b93b2857795ea8ce (table test-table [id=3f6c62153a7b4cb1922e4a3435adcbe3]) successfully deleted
I20260927 23:08:38.926342 22236 catalog_manager.cc:4841] TS e1db5d3e54cd4e3d974e5b70362e2a87 (127.3.240.195:36035): tablet 2cab533981e64446b93b2857795ea8ce (table test-table [id=3f6c62153a7b4cb1922e4a3435adcbe3]) successfully deleted
I20260927 23:08:39.012174 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 22313
I20260927 23:08:39.030622 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 22447
Sep 27 23:08:39 dist-test-slave-lmfp krb5kdc[22207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:08:39.047062 22722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:39.031474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.3.240.194:41109)
0927 23:08:39.031676 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:39.031682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:39.031701 (+ 19us) server_negotiation.cc:407] Connection header received
0927 23:08:39.031807 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:39.031813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:39.031898 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:39.032025 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:39.033346 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:39.034534 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:39.035768 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:39.036425 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:39.036561 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:39.040567 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:39.040582 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:39.040602 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:39.040644 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:39.044353 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:39.044995 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:39.045005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:39.045009 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:39.045152 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:39.045500 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:39.045504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:39.045508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:39.045957 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:39.046236 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:39.046435 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 23:08:39.046664 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 23:08:39.047869 22570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:39.156782 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 22447
I20260927 23:08:39.176055 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 22580
Sep 27 23:08:39 dist-test-slave-lmfp krb5kdc[22207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:08:39.191267 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:39.176927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.3.240.195:36035)
0927 23:08:39.177109 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:39.177118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:39.177193 (+ 75us) server_negotiation.cc:407] Connection header received
0927 23:08:39.177402 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:39.177405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:39.177465 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:39.177570 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:39.178950 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:39.179740 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:39.181117 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:39.181630 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:39.181755 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:39.186033 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:39.186047 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:39.186061 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:39.186094 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:39.188640 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:39.189234 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:39.189241 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:39.189243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:39.189378 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:39.189719 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:39.189724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:39.189727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:39.190076 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:39.190363 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:39.190662 (+ 299us) server_negotiation.cc:299] Negotiation successful
0927 23:08:39.190875 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:08:39.191891 22703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:39.275612 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 22580
I20260927 23:08:39.291886 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 22220
I20260927 23:08:39.292960 22282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:39.394940 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 22220
2026-09-27T23:08:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4169 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-27T23:08:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:39Z Disabled control of system clock
I20260927 23:08:39.466071 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42425
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:42425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 23:08:39.773193 22761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:39.773492 22761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:39.773600 22761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:39.779817 22761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:39.779927 22761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:39.780009 22761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:39.780095 22761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:39.780162 22761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:39.788055 22761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:42425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:39.789181 22761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:39.790709 22761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:39.801839 22766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:39.801743 22769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:39.801806 22767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:39.802421 22761 server_base.cc:1034] running on GCE node
I20260927 23:08:39.803133 22761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:39.804453 22761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:39.805655 22761 hybrid_clock.cc:648] HybridClock initialized: now 1790550519805619 us; error 28 us; skew 500 ppm
I20260927 23:08:39.805985 22761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:39.808103 22761 webserver.cc:460] Webserver started at http://127.3.240.254:42849/ using document root <none> and password file <none>
I20260927 23:08:39.808725 22761 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:39.808851 22761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:39.809172 22761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:39.812058 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "a959770f16bc4b4dab787f399e810b99"
format_stamp: "Formatted at 2026-09-27 23:08:39 on dist-test-slave-lmfp"
server_key: "9e57282f8498f91ad12d19dcc5f33b18"
server_key_iv: "34ced32c2c7238d53b8e4abfa0e53a45"
server_key_version: "encryptionkey@0"
I20260927 23:08:39.812793 22761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "a959770f16bc4b4dab787f399e810b99"
format_stamp: "Formatted at 2026-09-27 23:08:39 on dist-test-slave-lmfp"
server_key: "9e57282f8498f91ad12d19dcc5f33b18"
server_key_iv: "34ced32c2c7238d53b8e4abfa0e53a45"
server_key_version: "encryptionkey@0"
I20260927 23:08:39.817507 22761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260927 23:08:39.820523 22775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:39.821768 22761 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:08:39.821950 22761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "a959770f16bc4b4dab787f399e810b99"
format_stamp: "Formatted at 2026-09-27 23:08:39 on dist-test-slave-lmfp"
server_key: "9e57282f8498f91ad12d19dcc5f33b18"
server_key_iv: "34ced32c2c7238d53b8e4abfa0e53a45"
server_key_version: "encryptionkey@0"
I20260927 23:08:39.822148 22761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:39.846208 22761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:39.846931 22761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:39.847213 22761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:39.864223 22761 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:42425
I20260927 23:08:39.864239 22826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:42425 every 8 connection(s)
I20260927 23:08:39.865410 22761 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:39.868788 22827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:39.874948 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 22761
I20260927 23:08:39.875139 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:39.875589 4035 external_mini_cluster.cc:1428] Setting key b47d0205aeb2d330fb0733f6efd91132
I20260927 23:08:39.878466 22827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99: Bootstrap starting.
I20260927 23:08:39.881374 22827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:39.882335 22827 log.cc:824] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:39.884836 22827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99: No bootstrap required, opened a new log
I20260927 23:08:39.887733 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:39.877213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.3.240.254:42425)
0927 23:08:39.878070 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:39.878080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:39.878115 (+ 35us) server_negotiation.cc:407] Connection header received
0927 23:08:39.879753 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:39.879773 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:39.880185 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:39.880525 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:39.881029 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:39.882312 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:39.883685 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:39.884311 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:39.884510 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:39.885583 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:39.885591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:39.885603 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:39.886092 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:39.886166 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:39.886187 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:39.886318 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:39.886497 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 23:08:39.886764 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":281,"threads_started":1}
I20260927 23:08:39.892259 22827 raft_consensus.cc:348] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959770f16bc4b4dab787f399e810b99" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42425 } }
I20260927 23:08:39.892616 22827 raft_consensus.cc:374] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:39.892719 22827 raft_consensus.cc:729] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a959770f16bc4b4dab787f399e810b99, State: Initialized, Role: FOLLOWER
I20260927 23:08:39.893240 22827 consensus_queue.cc:260] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959770f16bc4b4dab787f399e810b99" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42425 } }
I20260927 23:08:39.893486 22827 raft_consensus.cc:388] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:39.893640 22827 raft_consensus.cc:482] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:39.893798 22827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:39.896289 22827 raft_consensus.cc:504] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959770f16bc4b4dab787f399e810b99" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42425 } }
I20260927 23:08:39.896791 22827 leader_election.cc:302] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a959770f16bc4b4dab787f399e810b99; no voters:
I20260927 23:08:39.897308 22827 leader_election.cc:288] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:39.897440 22832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:39.898914 22832 raft_consensus.cc:686] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [term 1 LEADER]: Becoming Leader. State: Replica: a959770f16bc4b4dab787f399e810b99, State: Running, Role: LEADER
I20260927 23:08:39.899467 22827 sys_catalog.cc:564] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:39.899412 22832 consensus_queue.cc:237] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959770f16bc4b4dab787f399e810b99" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42425 } }
I20260927 23:08:39.901952 22834 sys_catalog.cc:455] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a959770f16bc4b4dab787f399e810b99. Latest consensus state: current_term: 1 leader_uuid: "a959770f16bc4b4dab787f399e810b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959770f16bc4b4dab787f399e810b99" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42425 } } }
I20260927 23:08:39.901978 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a959770f16bc4b4dab787f399e810b99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a959770f16bc4b4dab787f399e810b99" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42425 } } }
I20260927 23:08:39.902534 22834 sys_catalog.cc:458] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:39.902596 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:39.904430 22838 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:39.909307 22838 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:39.917814 22838 catalog_manager.cc:1347] Generated new cluster ID: 2589c5f28556483287da02c71a7eb86b
I20260927 23:08:39.917909 22838 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:39.927258 22838 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:39.928324 22838 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:39.944679 22838 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99: Generated new TSK 0
I20260927 23:08:39.945439 22838 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:39.955081 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42425
--builtin_ntp_servers=127.3.240.212:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 23:08:40.268466 22851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:40.268688 22851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:40.268829 22851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:40.275492 22851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:40.275568 22851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:40.275744 22851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:08:40.283885 22851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:40.285003 22851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:40.286621 22851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:40.298377 22856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:40.298383 22859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:40.298362 22857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:40.299543 22851 server_base.cc:1034] running on GCE node
I20260927 23:08:40.300053 22851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:40.300887 22851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:40.302073 22851 hybrid_clock.cc:648] HybridClock initialized: now 1790550520302010 us; error 54 us; skew 500 ppm
I20260927 23:08:40.302314 22851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:40.304239 22851 webserver.cc:460] Webserver started at http://127.3.240.193:45317/ using document root <none> and password file <none>
I20260927 23:08:40.304780 22851 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:40.304855 22851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:40.305100 22851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:40.308056 22851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "2ddf1f28009a43c08a79fb0dba4921f6"
format_stamp: "Formatted at 2026-09-27 23:08:40 on dist-test-slave-lmfp"
server_key: "719da8272e099b4d6d92d539c3c90209"
server_key_iv: "2f904def7be80670739a8ebd027a49b0"
server_key_version: "encryptionkey@0"
I20260927 23:08:40.308795 22851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "2ddf1f28009a43c08a79fb0dba4921f6"
format_stamp: "Formatted at 2026-09-27 23:08:40 on dist-test-slave-lmfp"
server_key: "719da8272e099b4d6d92d539c3c90209"
server_key_iv: "2f904def7be80670739a8ebd027a49b0"
server_key_version: "encryptionkey@0"
I20260927 23:08:40.313469 22851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:08:40.316576 22865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:40.317688 22851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:08:40.317849 22851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "2ddf1f28009a43c08a79fb0dba4921f6"
format_stamp: "Formatted at 2026-09-27 23:08:40 on dist-test-slave-lmfp"
server_key: "719da8272e099b4d6d92d539c3c90209"
server_key_iv: "2f904def7be80670739a8ebd027a49b0"
server_key_version: "encryptionkey@0"
I20260927 23:08:40.318051 22851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:40.329983 22851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:40.330679 22851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:40.330971 22851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:40.331872 22851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:40.333228 22851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:40.333348 22851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:40.333473 22851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:40.333557 22851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:40.364892 22851 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:34323
I20260927 23:08:40.364938 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:34323 every 8 connection(s)
I20260927 23:08:40.366171 22851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:08:40.369571 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 22851
I20260927 23:08:40.369747 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:08:40.370167 4035 external_mini_cluster.cc:1428] Setting key 5bb7820d0423b16747b8ff13e9e32823
I20260927 23:08:40.372982 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42425
--builtin_ntp_servers=127.3.240.212:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 23:08:40.379084 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:40.368624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:33647 (local address 127.3.240.254:42425)
0927 23:08:40.368794 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:40.368799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:40.370326 (+ 1527us) server_negotiation.cc:407] Connection header received
0927 23:08:40.372286 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:40.372291 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:40.372369 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:40.372479 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:40.373560 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:40.374718 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:40.376246 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:40.376932 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:40.377070 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:40.378236 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:40.378244 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:40.378251 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:40.378274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:40.378311 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:40.378327 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:40.378414 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:40.378597 (+ 183us) server_negotiation.cc:299] Negotiation successful
0927 23:08:40.378695 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:08:40.379794 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:40.369133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42425 (local address 127.3.240.193:33647)
0927 23:08:40.370170 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:40.370213 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:40.372025 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:40.372665 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:40.372673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:40.372755 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:40.373380 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:40.373413 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:40.374857 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:40.374860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:40.376110 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:40.376122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:40.377121 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:40.377125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:40.377302 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:40.377966 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:40.377968 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:40.378003 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:40.378069 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:40.378415 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:40.378422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:40.378427 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 23:08:40.378547 (+ 120us) client_negotiation.cc:240] Negotiation successful
0927 23:08:40.378790 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":341,"threads_started":1}
I20260927 23:08:40.381175 22978 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42425
I20260927 23:08:40.381457 22978 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:40.382192 22978 heartbeater.cc:511] Master 127.3.240.254:42425 requested a full tablet report, sending...
I20260927 23:08:40.384095 22792 ts_manager.cc:194] Registered new tserver with Master: 2ddf1f28009a43c08a79fb0dba4921f6 (127.3.240.193:34323)
I20260927 23:08:40.385403 22792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.193:33647
W20260927 23:08:40.682668 22981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:40.682904 22981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:40.683068 22981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:40.689813 22981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:40.689898 22981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:40.689992 22981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:08:40.698388 22981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:40.699664 22981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:40.701435 22981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:40.713788 22986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:40.713732 22987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:40.713827 22989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:40.714538 22981 server_base.cc:1034] running on GCE node
I20260927 23:08:40.715189 22981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:40.716179 22981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:40.717414 22981 hybrid_clock.cc:648] HybridClock initialized: now 1790550520717379 us; error 44 us; skew 500 ppm
I20260927 23:08:40.717743 22981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:40.719848 22981 webserver.cc:460] Webserver started at http://127.3.240.194:34511/ using document root <none> and password file <none>
I20260927 23:08:40.720465 22981 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:40.720590 22981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:40.720885 22981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:40.723564 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "b8d58003ee7c4e798bd2e3b041f8821f"
format_stamp: "Formatted at 2026-09-27 23:08:40 on dist-test-slave-lmfp"
server_key: "9e33176aa2c4570aac1a6cce8a30551b"
server_key_iv: "8a391df0b1b8abe05892c52ab32759f1"
server_key_version: "encryptionkey@0"
I20260927 23:08:40.724289 22981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "b8d58003ee7c4e798bd2e3b041f8821f"
format_stamp: "Formatted at 2026-09-27 23:08:40 on dist-test-slave-lmfp"
server_key: "9e33176aa2c4570aac1a6cce8a30551b"
server_key_iv: "8a391df0b1b8abe05892c52ab32759f1"
server_key_version: "encryptionkey@0"
I20260927 23:08:40.728799 22981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:08:40.731792 22995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:40.732918 22981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:08:40.733100 22981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "b8d58003ee7c4e798bd2e3b041f8821f"
format_stamp: "Formatted at 2026-09-27 23:08:40 on dist-test-slave-lmfp"
server_key: "9e33176aa2c4570aac1a6cce8a30551b"
server_key_iv: "8a391df0b1b8abe05892c52ab32759f1"
server_key_version: "encryptionkey@0"
I20260927 23:08:40.733287 22981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:40.746696 22981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:40.747439 22981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:40.747726 22981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:40.748636 22981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:40.749892 22981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:40.750008 22981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:40.750129 22981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:40.750211 22981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:40.782017 22981 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:39857
I20260927 23:08:40.782096 23107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:39857 every 8 connection(s)
I20260927 23:08:40.783257 22981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:08:40.786922 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 22981
I20260927 23:08:40.787110 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:08:40.787592 4035 external_mini_cluster.cc:1428] Setting key b4193d4088ee7d20863046e4a01a7f31
I20260927 23:08:40.790028 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42425
--builtin_ntp_servers=127.3.240.212:43111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 23:08:40.795357 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:40.785712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:44753 (local address 127.3.240.254:42425)
0927 23:08:40.785909 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:40.785915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:40.787341 (+ 1426us) server_negotiation.cc:407] Connection header received
0927 23:08:40.789208 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:40.789213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:40.789292 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:40.789405 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:40.790460 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:40.791206 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:40.792723 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:40.793109 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:40.793238 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:40.794553 (+ 1315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:40.794558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:40.794561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:40.794577 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:40.794607 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:40.794618 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:40.794709 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:40.794928 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 23:08:40.795041 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:08:40.796281 23110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:40.786266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42425 (local address 127.3.240.194:44753)
0927 23:08:40.787176 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:40.787226 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:40.788937 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:40.789623 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:40.789631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:40.789711 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:40.790305 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:40.790322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:40.791353 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:40.791358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:40.792585 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:40.792597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:40.793315 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:40.793322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:40.793548 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:40.794262 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:40.794265 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:40.794300 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:40.794367 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:40.794708 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:40.794720 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:40.794730 (+ 10us) client_negotiation.cc:769] Sending connection context
0927 23:08:40.794900 (+ 170us) client_negotiation.cc:240] Negotiation successful
0927 23:08:40.795281 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":264,"threads_started":1}
I20260927 23:08:40.797596 23108 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42425
I20260927 23:08:40.797904 23108 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:40.798620 23108 heartbeater.cc:511] Master 127.3.240.254:42425 requested a full tablet report, sending...
I20260927 23:08:40.800073 22792 ts_manager.cc:194] Registered new tserver with Master: b8d58003ee7c4e798bd2e3b041f8821f (127.3.240.194:39857)
I20260927 23:08:40.801254 22792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.194:44753
W20260927 23:08:41.094514 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:41.094790 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:41.094870 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:41.101388 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:41.101500 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:41.101629 23111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:08:41.114717 23111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42425
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:41.115898 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:41.117555 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:41.130468 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:41.130605 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:41.130760 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:41.131201 23111 server_base.cc:1034] running on GCE node
I20260927 23:08:41.131775 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:41.132671 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:41.133904 23111 hybrid_clock.cc:648] HybridClock initialized: now 1790550521133868 us; error 57 us; skew 500 ppm
I20260927 23:08:41.134265 23111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:41.136598 23111 webserver.cc:460] Webserver started at http://127.3.240.195:44031/ using document root <none> and password file <none>
I20260927 23:08:41.137319 23111 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:41.137442 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:41.137768 23111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:41.140439 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "84399156cfbc4e79865067c29d86772d"
format_stamp: "Formatted at 2026-09-27 23:08:41 on dist-test-slave-lmfp"
server_key: "ce51e05219750772b8f267b0d8a8a257"
server_key_iv: "e55c74729ff2befed5eae793331c4d87"
server_key_version: "encryptionkey@0"
I20260927 23:08:41.141224 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "84399156cfbc4e79865067c29d86772d"
format_stamp: "Formatted at 2026-09-27 23:08:41 on dist-test-slave-lmfp"
server_key: "ce51e05219750772b8f267b0d8a8a257"
server_key_iv: "e55c74729ff2befed5eae793331c4d87"
server_key_version: "encryptionkey@0"
I20260927 23:08:41.145881 23111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 23:08:41.149469 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:41.150655 23111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 23:08:41.150810 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "84399156cfbc4e79865067c29d86772d"
format_stamp: "Formatted at 2026-09-27 23:08:41 on dist-test-slave-lmfp"
server_key: "ce51e05219750772b8f267b0d8a8a257"
server_key_iv: "e55c74729ff2befed5eae793331c4d87"
server_key_version: "encryptionkey@0"
I20260927 23:08:41.151021 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:41.171597 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:41.172514 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:41.172833 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:41.173848 23111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:41.175623 23111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:41.175765 23111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260927 23:08:41.175889 23111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:41.175982 23111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:41.212242 23111 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:35181
I20260927 23:08:41.212360 23237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:35181 every 8 connection(s)
I20260927 23:08:41.213644 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:08:41.222644 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 23111
I20260927 23:08:41.222842 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:08:41.223397 4035 external_mini_cluster.cc:1428] Setting key e47bca78335f2d5892d84d9af282887d
I20260927 23:08:41.226438 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.216096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:34919 (local address 127.3.240.254:42425)
0927 23:08:41.216305 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.216311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.217726 (+ 1415us) server_negotiation.cc:407] Connection header received
0927 23:08:41.219505 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.219510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.219578 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.219672 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:41.220690 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.221562 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.223424 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.224000 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.224147 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.225621 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:41.225626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:41.225630 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:41.225647 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:41.225678 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:41.225689 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:41.225783 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.226033 (+ 250us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.226122 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:08:41.227434 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.216705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42425 (local address 127.3.240.195:34919)
0927 23:08:41.217537 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.217573 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:41.219248 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:41.219843 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:41.219851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:41.219926 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:41.220522 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.220538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.221726 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.221729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.223252 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.223263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.224240 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.224248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.224496 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.225266 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:41.225272 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:41.225325 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:41.225434 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:41.225792 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:41.225804 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:41.225812 (+ 8us) client_negotiation.cc:769] Sending connection context
0927 23:08:41.225993 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 23:08:41.226362 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":247,"threads_started":1}
I20260927 23:08:41.228536 23238 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42425
I20260927 23:08:41.228773 23238 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:41.229403 23238 heartbeater.cc:511] Master 127.3.240.254:42425 requested a full tablet report, sending...
I20260927 23:08:41.230647 22792 ts_manager.cc:194] Registered new tserver with Master: 84399156cfbc4e79865067c29d86772d (127.3.240.195:35181)
I20260927 23:08:41.231385 22792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.195:34919
I20260927 23:08:41.238041 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:08:41.247241 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.242007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.3.240.254:42425)
0927 23:08:41.242154 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.242158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.242596 (+ 438us) server_negotiation.cc:407] Connection header received
0927 23:08:41.242717 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.242720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.242783 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.242853 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:41.243329 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.244001 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.245258 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.245643 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.245727 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.246628 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:41.246632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:41.246635 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:41.246650 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:41.246676 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:41.246705 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:41.246808 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.246947 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.246984 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:08:41.250459 22792 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:08:41.257506 22792 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:08:41.290302 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.269310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47616 (local address 127.3.240.194:39857)
0927 23:08:41.270298 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.270305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.280128 (+ 9823us) server_negotiation.cc:407] Connection header received
0927 23:08:41.280625 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.280634 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.280879 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.281056 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:41.282189 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.284105 (+ 1916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.287905 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.289181 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.289321 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.289570 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.289789 (+ 219us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.289935 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":666,"threads_started":1}
I20260927 23:08:41.290309 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.268955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:39857 (local address 127.0.0.1:47616)
0927 23:08:41.279978 (+ 11023us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.280001 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:41.280419 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:41.281420 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:41.281428 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:41.281472 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:41.281980 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.281995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.284334 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.284341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.286759 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.286776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.289380 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.289386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.289575 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.289615 (+ 40us) client_negotiation.cc:769] Sending connection context
0927 23:08:41.289728 (+ 113us) client_negotiation.cc:240] Negotiation successful
0927 23:08:41.289927 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10871,"thread_start_us":11209,"threads_started":1}
I20260927 23:08:41.294555 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.268511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:35181 (local address 127.0.0.1:57440)
0927 23:08:41.269407 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.269451 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:41.269650 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:41.288004 (+ 18354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:41.288008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:41.288029 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:41.288408 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.288422 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.289882 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.289886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.292223 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.292234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.293856 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.293863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.294022 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.294061 (+ 39us) client_negotiation.cc:769] Sending connection context
0927 23:08:41.294147 (+ 86us) client_negotiation.cc:240] Negotiation successful
0927 23:08:41.294265 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"mutex_wait_us":185,"thread_start_us":450,"threads_started":1}
I20260927 23:08:41.294070 23043 tablet_service.cc:1467] Processing CreateTablet for tablet 30c2fbaf9924423895fc932cea997448 (DEFAULT_TABLE table=test-table [id=e3c16bab0ea5410aae8355c64f3badce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:41.294934 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.270895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.0.0.1:59708)
0927 23:08:41.287663 (+ 16768us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.287676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:41.287762 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:41.288258 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:41.288262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:41.288289 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:41.288661 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.288672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.290326 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.290333 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.292526 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.292541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.294176 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.294182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.294345 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.294378 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 23:08:41.294503 (+ 125us) client_negotiation.cc:240] Negotiation successful
0927 23:08:41.294611 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16662,"mutex_wait_us":4,"thread_start_us":16533,"threads_started":1}
I20260927 23:08:41.295023 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.271179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.3.240.193:34323)
0927 23:08:41.287487 (+ 16308us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.287494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.287767 (+ 273us) server_negotiation.cc:407] Connection header received
0927 23:08:41.287861 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.287870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.288075 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.288236 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:41.288816 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.290134 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.292721 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.294018 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.294144 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.294340 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.294538 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.294681 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16147,"thread_start_us":16299,"threads_started":1}
I20260927 23:08:41.295456 23043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c2fbaf9924423895fc932cea997448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:41.295707 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.268805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.3.240.195:35181)
0927 23:08:41.287373 (+ 18568us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.287380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.287411 (+ 31us) server_negotiation.cc:407] Connection header received
0927 23:08:41.287530 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.287536 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.287768 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.287990 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:41.288605 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.289742 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.292403 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.293673 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.293807 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.294039 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.295233 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.295381 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18413,"thread_start_us":18252,"threads_started":1}
I20260927 23:08:41.298671 23173 tablet_service.cc:1467] Processing CreateTablet for tablet 30c2fbaf9924423895fc932cea997448 (DEFAULT_TABLE table=test-table [id=e3c16bab0ea5410aae8355c64f3badce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:41.299384 22913 tablet_service.cc:1467] Processing CreateTablet for tablet 30c2fbaf9924423895fc932cea997448 (DEFAULT_TABLE table=test-table [id=e3c16bab0ea5410aae8355c64f3badce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:41.300145 23173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c2fbaf9924423895fc932cea997448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:41.300755 22913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30c2fbaf9924423895fc932cea997448. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:41.309773 23256 tablet_bootstrap.cc:492] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: Bootstrap starting.
I20260927 23:08:41.311645 23257 tablet_bootstrap.cc:492] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: Bootstrap starting.
I20260927 23:08:41.313146 23256 tablet_bootstrap.cc:654] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:41.314301 23256 log.cc:824] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:41.314663 23258 tablet_bootstrap.cc:492] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6: Bootstrap starting.
I20260927 23:08:41.315130 23257 tablet_bootstrap.cc:654] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:41.316305 23257 log.cc:824] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:41.317728 23256 tablet_bootstrap.cc:492] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: No bootstrap required, opened a new log
I20260927 23:08:41.317937 23258 tablet_bootstrap.cc:654] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:41.318030 23256 ts_tablet_manager.cc:1397] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260927 23:08:41.319057 23257 tablet_bootstrap.cc:492] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: No bootstrap required, opened a new log
I20260927 23:08:41.319108 23258 log.cc:824] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:41.319335 23257 ts_tablet_manager.cc:1397] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260927 23:08:41.321929 23258 tablet_bootstrap.cc:492] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6: No bootstrap required, opened a new log
I20260927 23:08:41.322173 23258 ts_tablet_manager.cc:1397] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 23:08:41.327582 23257 raft_consensus.cc:348] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.327922 23257 raft_consensus.cc:374] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:41.328023 23257 raft_consensus.cc:729] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84399156cfbc4e79865067c29d86772d, State: Initialized, Role: FOLLOWER
I20260927 23:08:41.328158 23256 raft_consensus.cc:348] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.328572 23256 raft_consensus.cc:374] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:41.328676 23256 raft_consensus.cc:729] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8d58003ee7c4e798bd2e3b041f8821f, State: Initialized, Role: FOLLOWER
I20260927 23:08:41.328639 23257 consensus_queue.cc:260] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.329304 23256 consensus_queue.cc:260] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.329811 23238 heartbeater.cc:503] Master 127.3.240.254:42425 was elected leader, sending a full tablet report...
I20260927 23:08:41.330664 23257 ts_tablet_manager.cc:1428] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20260927 23:08:41.331174 23256 ts_tablet_manager.cc:1428] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260927 23:08:41.331238 23108 heartbeater.cc:503] Master 127.3.240.254:42425 was elected leader, sending a full tablet report...
I20260927 23:08:41.334075 23258 raft_consensus.cc:348] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.334563 23258 raft_consensus.cc:374] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:41.334689 23258 raft_consensus.cc:729] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ddf1f28009a43c08a79fb0dba4921f6, State: Initialized, Role: FOLLOWER
I20260927 23:08:41.335455 23258 consensus_queue.cc:260] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.336566 22978 heartbeater.cc:503] Master 127.3.240.254:42425 was elected leader, sending a full tablet report...
I20260927 23:08:41.337277 23258 ts_tablet_manager.cc:1428] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260927 23:08:41.368289 22979 tablet.cc:2369] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:41.393854 23263 raft_consensus.cc:482] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:08:41.394157 23263 raft_consensus.cc:504] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.395718 23263 leader_election.cc:288] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84399156cfbc4e79865067c29d86772d (127.3.240.195:35181), 2ddf1f28009a43c08a79fb0dba4921f6 (127.3.240.193:34323)
I20260927 23:08:41.402459 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.396211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:41515 (local address 127.3.240.195:35181)
0927 23:08:41.396395 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.396401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.397316 (+ 915us) server_negotiation.cc:407] Connection header received
0927 23:08:41.397484 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.397490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.397554 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.397618 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:41.398163 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.399123 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.400889 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.401672 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.401830 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.401895 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.402138 (+ 243us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.402250 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260927 23:08:41.402463 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.396207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:48367 (local address 127.3.240.193:34323)
0927 23:08:41.396395 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.396402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.396928 (+ 526us) server_negotiation.cc:407] Connection header received
0927 23:08:41.397167 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.397172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.397236 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.397330 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:41.397883 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.398818 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.400798 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.401550 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.401666 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.401713 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.402088 (+ 375us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.402216 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260927 23:08:41.402534 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.396127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.3.240.194:48367)
0927 23:08:41.396797 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.396814 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:41.397011 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:41.397361 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:41.397366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:41.397388 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:41.397728 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.397740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.398981 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.399022 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.400627 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.400638 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.401737 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.401746 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.401927 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.401962 (+ 35us) client_negotiation.cc:769] Sending connection context
0927 23:08:41.402031 (+ 69us) client_negotiation.cc:240] Negotiation successful
0927 23:08:41.402154 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":67,"thread_start_us":418,"threads_started":1}
I20260927 23:08:41.402618 23266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.396136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:35181 (local address 127.3.240.194:41515)
0927 23:08:41.397216 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.397229 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:41.397354 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:41.397679 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:41.397686 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:41.397717 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:41.398030 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.398038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.399269 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.399273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.400692 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:41.400703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.401852 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.401856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:41.401970 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.401992 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 23:08:41.402101 (+ 109us) client_negotiation.cc:240] Negotiation successful
0927 23:08:41.402211 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":52,"spinlock_wait_cycles":18176,"thread_start_us":771,"threads_started":1}
I20260927 23:08:41.403668 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c2fbaf9924423895fc932cea997448" candidate_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" is_pre_election: true
I20260927 23:08:41.403679 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c2fbaf9924423895fc932cea997448" candidate_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84399156cfbc4e79865067c29d86772d" is_pre_election: true
I20260927 23:08:41.404119 22933 raft_consensus.cc:2454] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d58003ee7c4e798bd2e3b041f8821f in term 0.
I20260927 23:08:41.404155 23193 raft_consensus.cc:2454] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8d58003ee7c4e798bd2e3b041f8821f in term 0.
I20260927 23:08:41.404790 22996 leader_election.cc:302] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ddf1f28009a43c08a79fb0dba4921f6, b8d58003ee7c4e798bd2e3b041f8821f; no voters:
I20260927 23:08:41.405126 23263 raft_consensus.cc:2781] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:08:41.405259 23263 raft_consensus.cc:482] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:08:41.405361 23263 raft_consensus.cc:3037] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:41.407421 23263 raft_consensus.cc:504] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.407939 23263 leader_election.cc:288] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [CANDIDATE]: Term 1 election: Requested vote from peers 84399156cfbc4e79865067c29d86772d (127.3.240.195:35181), 2ddf1f28009a43c08a79fb0dba4921f6 (127.3.240.193:34323)
I20260927 23:08:41.408317 23193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c2fbaf9924423895fc932cea997448" candidate_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84399156cfbc4e79865067c29d86772d"
I20260927 23:08:41.408361 22933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30c2fbaf9924423895fc932cea997448" candidate_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ddf1f28009a43c08a79fb0dba4921f6"
I20260927 23:08:41.408486 23193 raft_consensus.cc:3037] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:41.408555 22933 raft_consensus.cc:3037] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:41.410440 23193 raft_consensus.cc:2454] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d58003ee7c4e798bd2e3b041f8821f in term 1.
I20260927 23:08:41.410576 22933 raft_consensus.cc:2454] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8d58003ee7c4e798bd2e3b041f8821f in term 1.
I20260927 23:08:41.410796 22998 leader_election.cc:302] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84399156cfbc4e79865067c29d86772d, b8d58003ee7c4e798bd2e3b041f8821f; no voters:
I20260927 23:08:41.411113 23263 raft_consensus.cc:2781] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:41.411471 23263 raft_consensus.cc:686] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 1 LEADER]: Becoming Leader. State: Replica: b8d58003ee7c4e798bd2e3b041f8821f, State: Running, Role: LEADER
I20260927 23:08:41.411867 23263 consensus_queue.cc:237] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } }
I20260927 23:08:41.415911 22792 catalog_manager.cc:5495] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f reported cstate change: term changed from 0 to 1, leader changed from <none> to b8d58003ee7c4e798bd2e3b041f8821f (127.3.240.194). New cstate: current_term: 1 leader_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8d58003ee7c4e798bd2e3b041f8821f" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 } health_report { overall_health: UNKNOWN } } }
I20260927 23:08:41.432636 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.427835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.3.240.194:39857)
0927 23:08:41.427991 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.427996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.428012 (+ 16us) server_negotiation.cc:407] Connection header received
0927 23:08:41.428180 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.428183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.428247 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.428353 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:08:41.428685 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.429348 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.430584 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.430965 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.431122 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.431548 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:08:41.432030 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:08:41.432148 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.432310 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.432369 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:08:41.440879 22933 raft_consensus.cc:1261] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 1 FOLLOWER]: Refusing update from remote peer b8d58003ee7c4e798bd2e3b041f8821f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:41.440879 23193 raft_consensus.cc:1261] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 1 FOLLOWER]: Refusing update from remote peer b8d58003ee7c4e798bd2e3b041f8821f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:41.441980 23267 consensus_queue.cc:1035] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ddf1f28009a43c08a79fb0dba4921f6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 34323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:41.442400 23263 consensus_queue.cc:1035] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [LEADER]: Connected to new peer: Peer: permanent_uuid: "84399156cfbc4e79865067c29d86772d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 35181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:41.467159 22792 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:41.467409 22792 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:41.470566 22792 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a959770f16bc4b4dab787f399e810b99: Sending DeleteTablet for 3 replicas of tablet 30c2fbaf9924423895fc932cea997448
I20260927 23:08:41.471604 23173 tablet_service.cc:1514] Processing DeleteTablet for tablet 30c2fbaf9924423895fc932cea997448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:41 UTC) from {username='slave'} at 127.0.0.1:57440
I20260927 23:08:41.471609 23043 tablet_service.cc:1514] Processing DeleteTablet for tablet 30c2fbaf9924423895fc932cea997448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:41 UTC) from {username='slave'} at 127.0.0.1:47616
I20260927 23:08:41.472464 22913 tablet_service.cc:1514] Processing DeleteTablet for tablet 30c2fbaf9924423895fc932cea997448 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:41 UTC) from {username='slave'} at 127.0.0.1:59708
I20260927 23:08:41.472744 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 22851
I20260927 23:08:41.484822 23281 tablet_replica.cc:330] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: stopping tablet replica
I20260927 23:08:41.485463 23280 tablet_replica.cc:330] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: stopping tablet replica
I20260927 23:08:41.486218 23280 raft_consensus.cc:2229] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:41.486819 23281 raft_consensus.cc:2229] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:08:41.487682 23281 raft_consensus.cc:2258] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:41.488349 23280 raft_consensus.cc:2258] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:41.489118 23255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.474160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59720 (local address 127.3.240.193:34323)
0927 23:08:41.474333 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.474339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.480706 (+ 6367us) server_negotiation.cc:407] Connection header received
0927 23:08:41.480897 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.480903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.481016 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.481124 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:41.481801 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.483094 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.485094 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.485773 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.485871 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.487381 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:41.487393 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:41.487409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:41.488201 (+ 792us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:41.488261 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:41.488286 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:41.488370 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.488630 (+ 260us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.488742 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:08:41.490084 23282 tablet_replica.cc:330] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6: stopping tablet replica
I20260927 23:08:41.490446 22973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:41.490830 23280 ts_tablet_manager.cc:1905] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:41.495784 23280 ts_tablet_manager.cc:1918] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:41.496029 23280 log.cc:1196] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/30c2fbaf9924423895fc932cea997448
I20260927 23:08:41.496609 23280 ts_tablet_manager.cc:1939] T 30c2fbaf9924423895fc932cea997448 P 84399156cfbc4e79865067c29d86772d: Deleting consensus metadata
I20260927 23:08:41.497586 23281 ts_tablet_manager.cc:1905] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:41.499078 22778 catalog_manager.cc:4841] TS 84399156cfbc4e79865067c29d86772d (127.3.240.195:35181): tablet 30c2fbaf9924423895fc932cea997448 (table test-table [id=e3c16bab0ea5410aae8355c64f3badce]) successfully deleted
I20260927 23:08:41.505574 23281 ts_tablet_manager.cc:1918] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:41.505752 23281 log.cc:1196] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/30c2fbaf9924423895fc932cea997448
I20260927 23:08:41.506356 23281 ts_tablet_manager.cc:1939] T 30c2fbaf9924423895fc932cea997448 P b8d58003ee7c4e798bd2e3b041f8821f: Deleting consensus metadata
I20260927 23:08:41.508414 22779 catalog_manager.cc:4841] TS b8d58003ee7c4e798bd2e3b041f8821f (127.3.240.194:39857): tablet 30c2fbaf9924423895fc932cea997448 (table test-table [id=e3c16bab0ea5410aae8355c64f3badce]) successfully deleted
I20260927 23:08:41.490568 23282 raft_consensus.cc:2229] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:41.585436 23282 raft_consensus.cc:2258] T 30c2fbaf9924423895fc932cea997448 P 2ddf1f28009a43c08a79fb0dba4921f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:41.585628 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 22851
W20260927 23:08:41.609875 22776 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:34323 (error 108)
I20260927 23:08:41.610116 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 22981
I20260927 23:08:41.612254 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.611806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.0.0.1:59722)
0927 23:08:41.611973 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.612091 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:34323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260927 23:08:41.612533 22776 catalog_manager.cc:4584] TS 2ddf1f28009a43c08a79fb0dba4921f6 (127.3.240.193:34323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 30c2fbaf9924423895fc932cea997448: Network error: Client connection negotiation failed: client connection to 127.3.240.193:34323: connect: Connection refused (error 111)
I20260927 23:08:41.617192 23253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.611037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47630 (local address 127.3.240.194:39857)
0927 23:08:41.611295 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.611301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.611319 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:08:41.611443 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.611449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.611546 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.611649 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:41.612108 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.613052 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.614265 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.614658 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.614786 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.615906 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:41.615914 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:41.615927 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:41.616496 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:41.616539 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:41.616565 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:41.616681 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.616803 (+ 122us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.616898 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260927 23:08:41.618125 23103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:41.666236 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.665712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.0.0.1:59736)
0927 23:08:41.665929 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.666044 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:34323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260927 23:08:41.667274 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.666867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.0.0.1:59744)
0927 23:08:41.667049 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.667127 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:34323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260927 23:08:41.724969 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 22981
I20260927 23:08:41.728024 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.727509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.0.0.1:59754)
0927 23:08:41.727699 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.727779 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:34323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 23:08:41.728924 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.728623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.0.0.1:59762)
0927 23:08:41.728753 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.728820 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:34323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 23:08:41.743124 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 23111
I20260927 23:08:41.749416 23251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.743859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.3.240.195:35181)
0927 23:08:41.744027 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:41.744034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:41.744056 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:08:41.744265 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:41.744269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:41.744357 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:41.744458 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:41.744841 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.745539 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.746693 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:41.747127 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:41.747252 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:41.748193 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:41.748201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:41.748216 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:41.748738 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:41.748779 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:41.748796 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:41.748918 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:41.749019 (+ 101us) server_negotiation.cc:299] Negotiation successful
0927 23:08:41.749141 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:08:41.749985 23233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:41.809165 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.808745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.0.0.1:59774)
0927 23:08:41.808940 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.809027 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:34323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 23:08:41.810168 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:41.809780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:34323 (local address 127.0.0.1:59778)
0927 23:08:41.809996 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:41.810066 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:34323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260927 23:08:41.833647 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 23111
I20260927 23:08:41.849620 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 22761
I20260927 23:08:41.850632 22822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:41.956132 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 22761
2026-09-27T23:08:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2556 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:42 dist-test-slave-lmfp krb5kdc[23291](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:42 dist-test-slave-lmfp krb5kdc[23291](info): set up 2 sockets
Sep 27 23:08:42 dist-test-slave-lmfp krb5kdc[23291](info): commencing operation
krb5kdc: starting...
W20260927 23:08:42.987262 4035 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:43 dist-test-slave-lmfp krb5kdc[23291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:08:43.222514 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41151
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:41151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:43.538270 23304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:43.538551 23304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:43.538724 23304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:43.546128 23304 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:43.546216 23304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:43.546296 23304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:43.546379 23304 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:43.546469 23304 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:43.555775 23304 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:41151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:43.557051 23304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:43.558342 23304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:43.569994 23310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:43.570057 23309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:43.570086 23312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:43.571277 23304 server_base.cc:1034] running on GCE node
I20260927 23:08:43.572057 23304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:43.573424 23304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:43.574647 23304 hybrid_clock.cc:648] HybridClock initialized: now 1790550523574587 us; error 44 us; skew 500 ppm
I20260927 23:08:43.574972 23304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:43 dist-test-slave-lmfp krb5kdc[23291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550523, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:43.581264 23304 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:08:43.582613 23304 webserver.cc:460] Webserver started at http://127.3.240.254:41377/ using document root <none> and password file <none>
I20260927 23:08:43.583338 23304 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:43.583480 23304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:43.583771 23304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:43.586627 23304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "afd20723637846cca53e049e13f37572"
format_stamp: "Formatted at 2026-09-27 23:08:43 on dist-test-slave-lmfp"
server_key: "5de9e3de1e790f957de4f46477375d16"
server_key_iv: "9f37a62bb27c0e1b917091a1402a4222"
server_key_version: "encryptionkey@0"
I20260927 23:08:43.587373 23304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "afd20723637846cca53e049e13f37572"
format_stamp: "Formatted at 2026-09-27 23:08:43 on dist-test-slave-lmfp"
server_key: "5de9e3de1e790f957de4f46477375d16"
server_key_iv: "9f37a62bb27c0e1b917091a1402a4222"
server_key_version: "encryptionkey@0"
I20260927 23:08:43.592240 23304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 23:08:43.595953 23319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:43.597170 23304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:08:43.597350 23304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "afd20723637846cca53e049e13f37572"
format_stamp: "Formatted at 2026-09-27 23:08:43 on dist-test-slave-lmfp"
server_key: "5de9e3de1e790f957de4f46477375d16"
server_key_iv: "9f37a62bb27c0e1b917091a1402a4222"
server_key_version: "encryptionkey@0"
I20260927 23:08:43.597540 23304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:43.615188 23304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:43.618808 23304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:43.619138 23304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:43.638264 23304 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:41151
I20260927 23:08:43.638286 23370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:41151 every 8 connection(s)
I20260927 23:08:43.639678 23304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:43.643270 23371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:43.644634 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 23304
I20260927 23:08:43.644817 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:43.645213 4035 external_mini_cluster.cc:1428] Setting key 77c3c9f4345325bf57cede4e5d1d773c
I20260927 23:08:43.657033 23371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572: Bootstrap starting.
Sep 27 23:08:43 dist-test-slave-lmfp krb5kdc[23291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:43.659933 23371 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:43.661046 23371 log.cc:824] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:43.663870 23371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572: No bootstrap required, opened a new log
I20260927 23:08:43.667824 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:43.647036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.3.240.254:41151)
0927 23:08:43.647774 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:43.647787 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:43.647832 (+ 45us) server_negotiation.cc:407] Connection header received
0927 23:08:43.649516 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:43.649543 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:43.650029 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:43.650458 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:43.651811 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:43.653810 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:43.655138 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:43.656112 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:43.656297 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:43.660183 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:43.660190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:43.660202 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:43.660247 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:43.663402 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:43.664117 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:43.664125 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:43.664132 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:43.664345 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:43.664738 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:43.664745 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:43.664748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:43.665401 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:43.665699 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:43.666053 (+ 354us) server_negotiation.cc:299] Negotiation successful
0927 23:08:43.666514 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":303,"threads_started":1}
I20260927 23:08:43.672384 23371 raft_consensus.cc:348] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd20723637846cca53e049e13f37572" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41151 } }
I20260927 23:08:43.672717 23371 raft_consensus.cc:374] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:43.672791 23371 raft_consensus.cc:729] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd20723637846cca53e049e13f37572, State: Initialized, Role: FOLLOWER
I20260927 23:08:43.673238 23371 consensus_queue.cc:260] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd20723637846cca53e049e13f37572" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41151 } }
I20260927 23:08:43.673430 23371 raft_consensus.cc:388] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:43.673529 23371 raft_consensus.cc:482] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:43.673650 23371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:43.675848 23371 raft_consensus.cc:504] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd20723637846cca53e049e13f37572" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41151 } }
I20260927 23:08:43.676308 23371 leader_election.cc:302] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: afd20723637846cca53e049e13f37572; no voters:
I20260927 23:08:43.676942 23371 leader_election.cc:288] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:43.677058 23376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:43.678071 23376 raft_consensus.cc:686] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [term 1 LEADER]: Becoming Leader. State: Replica: afd20723637846cca53e049e13f37572, State: Running, Role: LEADER
I20260927 23:08:43.678606 23376 consensus_queue.cc:237] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd20723637846cca53e049e13f37572" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41151 } }
I20260927 23:08:43.678927 23371 sys_catalog.cc:564] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:43.681420 23378 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader afd20723637846cca53e049e13f37572. Latest consensus state: current_term: 1 leader_uuid: "afd20723637846cca53e049e13f37572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd20723637846cca53e049e13f37572" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41151 } } }
I20260927 23:08:43.681375 23377 sys_catalog.cc:455] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "afd20723637846cca53e049e13f37572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "afd20723637846cca53e049e13f37572" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41151 } } }
I20260927 23:08:43.681802 23378 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:43.681874 23377 sys_catalog.cc:458] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:43.682673 23385 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:43.687305 23385 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:43.696171 23385 catalog_manager.cc:1347] Generated new cluster ID: 5ee958b2a2f646b38526bbad5facdc16
I20260927 23:08:43.696277 23385 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:43.725673 23385 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:43.727169 23385 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:43.736392 23385 catalog_manager.cc:5868] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572: Generated new TSK 0
I20260927 23:08:43.737346 23385 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260927 23:08:43.799131 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41151
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:44.123888 23397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:44.124166 23397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:44.124337 23397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:44.131558 23397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:44.131687 23397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:44.131878 23397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:08:44.141541 23397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:44.142788 23397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:44.144124 23397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:44.157383 23402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:44.157332 23403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:44.157413 23405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:44.158118 23397 server_base.cc:1034] running on GCE node
I20260927 23:08:44.158773 23397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:44.159731 23397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:44.160959 23397 hybrid_clock.cc:648] HybridClock initialized: now 1790550524160913 us; error 54 us; skew 500 ppm
I20260927 23:08:44.161281 23397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:44 dist-test-slave-lmfp krb5kdc[23291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:44.164315 23397 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260927 23:08:44.165675 23397 webserver.cc:460] Webserver started at http://127.3.240.193:37473/ using document root <none> and password file <none>
I20260927 23:08:44.166400 23397 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:44.166543 23397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:44.166846 23397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:44.170087 23397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "c40d155ca09f48638bf756c49c3c3af8"
format_stamp: "Formatted at 2026-09-27 23:08:44 on dist-test-slave-lmfp"
server_key: "d735e89713fb8776869004e41e40962f"
server_key_iv: "55247c44475548b94c7462eccbcf56d7"
server_key_version: "encryptionkey@0"
I20260927 23:08:44.170814 23397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "c40d155ca09f48638bf756c49c3c3af8"
format_stamp: "Formatted at 2026-09-27 23:08:44 on dist-test-slave-lmfp"
server_key: "d735e89713fb8776869004e41e40962f"
server_key_iv: "55247c44475548b94c7462eccbcf56d7"
server_key_version: "encryptionkey@0"
I20260927 23:08:44.175745 23397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 23:08:44.178933 23412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:44.180452 23397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:08:44.180613 23397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "c40d155ca09f48638bf756c49c3c3af8"
format_stamp: "Formatted at 2026-09-27 23:08:44 on dist-test-slave-lmfp"
server_key: "d735e89713fb8776869004e41e40962f"
server_key_iv: "55247c44475548b94c7462eccbcf56d7"
server_key_version: "encryptionkey@0"
I20260927 23:08:44.180809 23397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:44.223814 23397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:44.227607 23397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:44.227896 23397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:44.228816 23397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:44.230079 23397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:44.230180 23397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:44.230324 23397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:44.230394 23397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:44.262642 23397 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:46407
I20260927 23:08:44.262719 23524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:46407 every 8 connection(s)
I20260927 23:08:44.263825 23397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:08:44.266330 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 23397
I20260927 23:08:44.266511 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:08:44.267099 4035 external_mini_cluster.cc:1428] Setting key fd1fc2bd39d1ad5cacba2ece346abc05
Sep 27 23:08:44 dist-test-slave-lmfp krb5kdc[23291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:44.286155 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:44.266494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:35365 (local address 127.3.240.254:41151)
0927 23:08:44.266971 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:44.266978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:44.268072 (+ 1094us) server_negotiation.cc:407] Connection header received
0927 23:08:44.270493 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:44.270497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:44.270557 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:44.270635 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:44.272595 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:44.273443 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:44.276180 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:44.276594 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:44.276716 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:44.280655 (+ 3939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:44.280660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:44.280663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:44.280705 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:44.283257 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:44.283940 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:44.283949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:44.283954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:44.284087 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:44.284504 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:44.284509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:44.284514 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:44.284783 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:44.284938 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:44.285612 (+ 674us) server_negotiation.cc:299] Negotiation successful
0927 23:08:44.285817 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
I20260927 23:08:44.287132 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:44.267129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:41151 (local address 127.3.240.193:35365)
0927 23:08:44.267893 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:44.267947 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:44.270198 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:44.270894 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:44.270910 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:44.271678 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:44.272422 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:44.272445 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:44.273611 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:44.273619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:44.276014 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:44.276032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:44.276830 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:44.276839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:44.277148 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:44.277893 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:44.277924 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:44.280432 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:44.283399 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:44.283407 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:44.283423 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:44.283815 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:44.284191 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:44.284195 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:44.284199 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:44.284390 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:44.284913 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:44.284929 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:44.285238 (+ 309us) client_negotiation.cc:769] Sending connection context
0927 23:08:44.285485 (+ 247us) client_negotiation.cc:240] Negotiation successful
0927 23:08:44.285978 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":268,"threads_started":1}
I20260927 23:08:44.288717 23525 heartbeater.cc:347] Connected to a master server at 127.3.240.254:41151
I20260927 23:08:44.289067 23525 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:44.289952 23525 heartbeater.cc:511] Master 127.3.240.254:41151 requested a full tablet report, sending...
I20260927 23:08:44.292086 23336 ts_manager.cc:194] Registered new tserver with Master: c40d155ca09f48638bf756c49c3c3af8 (127.3.240.193:46407)
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260927 23:08:44.293608 23336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:35365
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260927 23:08:44.321938 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41151
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:44.633868 23531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:44.634114 23531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:44.634198 23531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:44.641383 23531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:44.641489 23531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:44.641690 23531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:08:44.651189 23531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:44.652294 23531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:44.653528 23531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:44.666182 23537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:44.666227 23536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:44.666654 23539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:44.667085 23531 server_base.cc:1034] running on GCE node
I20260927 23:08:44.667624 23531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:44.668468 23531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:44.669687 23531 hybrid_clock.cc:648] HybridClock initialized: now 1790550524669655 us; error 43 us; skew 500 ppm
I20260927 23:08:44.669983 23531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:44 dist-test-slave-lmfp krb5kdc[23291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:44.673053 23531 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260927 23:08:44.674376 23531 webserver.cc:460] Webserver started at http://127.3.240.194:43055/ using document root <none> and password file <none>
I20260927 23:08:44.675071 23531 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:44.675196 23531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:44.675501 23531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:44.678279 23531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "1eaee110076e452488804377de090a39"
format_stamp: "Formatted at 2026-09-27 23:08:44 on dist-test-slave-lmfp"
server_key: "eb7a72a429b422f5fe6bb73e2886b26b"
server_key_iv: "0be732a1dd8e20bb69d333199e9b89ac"
server_key_version: "encryptionkey@0"
I20260927 23:08:44.679090 23531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "1eaee110076e452488804377de090a39"
format_stamp: "Formatted at 2026-09-27 23:08:44 on dist-test-slave-lmfp"
server_key: "eb7a72a429b422f5fe6bb73e2886b26b"
server_key_iv: "0be732a1dd8e20bb69d333199e9b89ac"
server_key_version: "encryptionkey@0"
I20260927 23:08:44.683866 23531 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260927 23:08:44.686972 23546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:44.688344 23531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:08:44.688522 23531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "1eaee110076e452488804377de090a39"
format_stamp: "Formatted at 2026-09-27 23:08:44 on dist-test-slave-lmfp"
server_key: "eb7a72a429b422f5fe6bb73e2886b26b"
server_key_iv: "0be732a1dd8e20bb69d333199e9b89ac"
server_key_version: "encryptionkey@0"
I20260927 23:08:44.688720 23531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:44.704264 23531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:44.708762 23531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:44.709074 23531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:44.710023 23531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:44.711432 23531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:44.711535 23531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:44.711697 23531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:44.711777 23531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:44.744932 23531 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:45955
I20260927 23:08:44.744993 23658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:45955 every 8 connection(s)
I20260927 23:08:44.746286 23531 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:08:44.752305 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 23531
I20260927 23:08:44.752501 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:08:44.752947 4035 external_mini_cluster.cc:1428] Setting key c150588e039e08dfd4419d1402ac9841
Sep 27 23:08:44 dist-test-slave-lmfp krb5kdc[23291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550524, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:44.765882 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:44.748783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:55913 (local address 127.3.240.254:41151)
0927 23:08:44.748966 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:44.748970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:44.750143 (+ 1173us) server_negotiation.cc:407] Connection header received
0927 23:08:44.751950 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:44.751956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:44.752052 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:44.752162 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:44.753813 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:44.754501 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:44.756099 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:44.756560 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:44.756705 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:44.760014 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:44.760019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:44.760022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:44.760070 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:44.763031 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:44.763684 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:44.763687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:44.763690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:44.763849 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:44.764261 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:44.764265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:44.764268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:44.764547 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:44.764695 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:44.765325 (+ 630us) server_negotiation.cc:299] Negotiation successful
0927 23:08:44.765560 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 23:08:44.766880 23661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:44.749193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:41151 (local address 127.3.240.194:55913)
0927 23:08:44.749953 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:44.750013 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:44.751704 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:44.752384 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:44.752412 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:44.753078 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:44.753634 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:44.753658 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:44.754674 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:44.754678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:44.755953 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:44.755965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:44.756776 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:44.756782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:44.757015 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:44.757679 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:44.757699 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:44.759808 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:44.763155 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:44.763161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:44.763172 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:44.763578 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:44.763953 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:44.763957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:44.763961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:44.764138 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:44.764670 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:44.764677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:44.764991 (+ 314us) client_negotiation.cc:769] Sending connection context
0927 23:08:44.765232 (+ 241us) client_negotiation.cc:240] Negotiation successful
0927 23:08:44.765690 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":364,"threads_started":1}
I20260927 23:08:44.768329 23659 heartbeater.cc:347] Connected to a master server at 127.3.240.254:41151
I20260927 23:08:44.768604 23659 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:44.769256 23659 heartbeater.cc:511] Master 127.3.240.254:41151 requested a full tablet report, sending...
I20260927 23:08:44.770663 23336 ts_manager.cc:194] Registered new tserver with Master: 1eaee110076e452488804377de090a39 (127.3.240.194:45955)
I20260927 23:08:44.771497 23336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:55913
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260927 23:08:44.805446 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41151
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:45.123410 23664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:45.123693 23664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:45.123893 23664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:45.131209 23664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:45.131331 23664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:45.131559 23664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:08:45.141345 23664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41151
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:45.142634 23664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:45.143996 23664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:45.156947 23670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:45.157037 23669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:45.157104 23672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:45.158025 23664 server_base.cc:1034] running on GCE node
I20260927 23:08:45.158600 23664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:45.159525 23664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:45.160717 23664 hybrid_clock.cc:648] HybridClock initialized: now 1790550525160670 us; error 36 us; skew 500 ppm
I20260927 23:08:45.160966 23664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:45 dist-test-slave-lmfp krb5kdc[23291](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:45.164088 23664 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260927 23:08:45.165381 23664 webserver.cc:460] Webserver started at http://127.3.240.195:36343/ using document root <none> and password file <none>
I20260927 23:08:45.166006 23664 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:45.166098 23664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:45.166342 23664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:45.169102 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "d4da5e595d4246de9a7fea446c0d1b7c"
format_stamp: "Formatted at 2026-09-27 23:08:45 on dist-test-slave-lmfp"
server_key: "1d53511c93c7ea8c789e22b9e21a18d6"
server_key_iv: "99e8851c1fdd1deb43603211f64f42d7"
server_key_version: "encryptionkey@0"
I20260927 23:08:45.169775 23664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "d4da5e595d4246de9a7fea446c0d1b7c"
format_stamp: "Formatted at 2026-09-27 23:08:45 on dist-test-slave-lmfp"
server_key: "1d53511c93c7ea8c789e22b9e21a18d6"
server_key_iv: "99e8851c1fdd1deb43603211f64f42d7"
server_key_version: "encryptionkey@0"
I20260927 23:08:45.174911 23664 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 23:08:45.178153 23679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:45.179531 23664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260927 23:08:45.179695 23664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "d4da5e595d4246de9a7fea446c0d1b7c"
format_stamp: "Formatted at 2026-09-27 23:08:45 on dist-test-slave-lmfp"
server_key: "1d53511c93c7ea8c789e22b9e21a18d6"
server_key_iv: "99e8851c1fdd1deb43603211f64f42d7"
server_key_version: "encryptionkey@0"
I20260927 23:08:45.179884 23664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:45.203614 23664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:45.207401 23664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:45.207688 23664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:45.208599 23664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:45.209841 23664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:45.209940 23664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:45.210098 23664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:45.210165 23664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:45.242324 23664 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:34315
I20260927 23:08:45.242408 23791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:34315 every 8 connection(s)
I20260927 23:08:45.243642 23664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:08:45.245191 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 23664
I20260927 23:08:45.245364 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:08:45.245873 4035 external_mini_cluster.cc:1428] Setting key 37797b36b9edc0a652b40893c83032fc
Sep 27 23:08:45 dist-test-slave-lmfp krb5kdc[23291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550525, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:08:45.263212 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.246368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:57187 (local address 127.3.240.254:41151)
0927 23:08:45.246617 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.246623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.248048 (+ 1425us) server_negotiation.cc:407] Connection header received
0927 23:08:45.250121 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.250125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.250187 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.250261 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:45.251712 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.252449 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.253890 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.254265 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.254352 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.257518 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:45.257523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:45.257526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:45.257556 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:45.260208 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.260892 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.260898 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.260903 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.261091 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.261515 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.261519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.261521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.261774 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:45.261920 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.262618 (+ 698us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.262834 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260927 23:08:45.263960 23794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.247020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:41151 (local address 127.3.240.195:57187)
0927 23:08:45.247852 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:45.247919 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:45.249847 (+ 1928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:45.250458 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:45.250466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:45.250947 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:45.251562 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.251579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.252561 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.252564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.253768 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.253780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.254430 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.254435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.254618 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.255270 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:08:45.255289 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:45.257339 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:45.260345 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:45.260354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:45.260368 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:45.260766 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:45.261226 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:08:45.261229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:08:45.261231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:08:45.261402 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:08:45.261882 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:45.261887 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:45.262194 (+ 307us) client_negotiation.cc:769] Sending connection context
0927 23:08:45.262459 (+ 265us) client_negotiation.cc:240] Negotiation successful
0927 23:08:45.262828 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":349,"threads_started":1}
I20260927 23:08:45.265226 23792 heartbeater.cc:347] Connected to a master server at 127.3.240.254:41151
I20260927 23:08:45.265547 23792 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:45.266327 23792 heartbeater.cc:511] Master 127.3.240.254:41151 requested a full tablet report, sending...
I20260927 23:08:45.267890 23336 ts_manager.cc:194] Registered new tserver with Master: d4da5e595d4246de9a7fea446c0d1b7c (127.3.240.195:34315)
I20260927 23:08:45.268675 23336 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:57187
I20260927 23:08:45.272830 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:08:45.288811 23527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.276762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.3.240.254:41151)
0927 23:08:45.276909 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.276914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.277358 (+ 444us) server_negotiation.cc:407] Connection header received
0927 23:08:45.277477 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.277486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.277538 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.277614 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:45.278833 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.279685 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.281002 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.281448 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.281546 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.284015 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:45.284020 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:45.284023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:45.284051 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:45.286566 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.287200 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.287204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.287206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.287321 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.287632 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.287636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.287638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.287929 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:45.288100 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.288367 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.288510 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 23:08:45.296671 23525 heartbeater.cc:503] Master 127.3.240.254:41151 was elected leader, sending a full tablet report...
I20260927 23:08:45.292454 23336 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60786:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:08:45.299707 23336 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:08:45.325496 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.311049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34315 (local address 127.0.0.1:39020)
0927 23:08:45.312639 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:45.312661 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:45.312789 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:45.314892 (+ 2103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:45.314903 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:45.314953 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:45.315431 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.315447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.320804 (+ 5357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.320809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.322922 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.322934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.324370 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.324377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.324579 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.324634 (+ 55us) client_negotiation.cc:769] Sending connection context
0927 23:08:45.324763 (+ 129us) client_negotiation.cc:240] Negotiation successful
0927 23:08:45.324954 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428,"mutex_wait_us":1,"thread_start_us":2232,"threads_started":1}
I20260927 23:08:45.326371 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.311308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.3.240.195:34315)
0927 23:08:45.313474 (+ 2166us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.313487 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.313527 (+ 40us) server_negotiation.cc:407] Connection header received
0927 23:08:45.313646 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.313657 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.313990 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.314480 (+ 490us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:45.315684 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.320639 (+ 4955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.323122 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.324161 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.324295 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.324532 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.325980 (+ 1448us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.326158 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1924,"thread_start_us":1753,"threads_started":1}
I20260927 23:08:45.327389 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.310745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:45955 (local address 127.0.0.1:53250)
0927 23:08:45.311689 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:45.311736 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:45.311950 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:45.320045 (+ 8095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:45.320051 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:45.320081 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:45.320558 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.320571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.322435 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.322440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.324665 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.324680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.326673 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.326679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.326843 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.326877 (+ 34us) client_negotiation.cc:769] Sending connection context
0927 23:08:45.326939 (+ 62us) client_negotiation.cc:240] Negotiation successful
0927 23:08:45.327088 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":736,"threads_started":1}
I20260927 23:08:45.328403 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.310956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.3.240.194:45955)
0927 23:08:45.319447 (+ 8491us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.319458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.319487 (+ 29us) server_negotiation.cc:407] Connection header received
0927 23:08:45.319592 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.319601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.319842 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.320217 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:45.320735 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.322292 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.324985 (+ 2693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.326452 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.327744 (+ 1292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.327899 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.327955 (+ 56us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.328107 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8344,"thread_start_us":8185,"threads_started":1}
I20260927 23:08:45.328596 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.313124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55178 (local address 127.3.240.193:46407)
0927 23:08:45.319843 (+ 6719us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.319851 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.320453 (+ 602us) server_negotiation.cc:407] Connection header received
0927 23:08:45.320549 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.320848 (+ 299us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.321093 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.321295 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:45.321855 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.323209 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.325952 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.327378 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.327521 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.327748 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.328103 (+ 355us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.328255 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6546,"thread_start_us":6415,"threads_started":1}
I20260927 23:08:45.328775 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.312421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:46407 (local address 127.0.0.1:55178)
0927 23:08:45.320268 (+ 7847us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:45.320289 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:45.320406 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:45.321285 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:45.321289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:45.321314 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:45.321696 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.321704 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.323364 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.323370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.325782 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.325795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.327755 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.327761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.327915 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.327953 (+ 38us) client_negotiation.cc:769] Sending connection context
0927 23:08:45.328047 (+ 94us) client_negotiation.cc:240] Negotiation successful
0927 23:08:45.328178 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7725,"thread_start_us":7545,"threads_started":1}
I20260927 23:08:45.330310 23727 tablet_service.cc:1467] Processing CreateTablet for tablet 90efce1de3cc44b9a9640ae62b4f5ee0 (DEFAULT_TABLE table=test-table [id=be9002c64d8f4417b163b5ab7f466677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:45.331804 23727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90efce1de3cc44b9a9640ae62b4f5ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:45.331655 23460 tablet_service.cc:1467] Processing CreateTablet for tablet 90efce1de3cc44b9a9640ae62b4f5ee0 (DEFAULT_TABLE table=test-table [id=be9002c64d8f4417b163b5ab7f466677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:45.331786 23594 tablet_service.cc:1467] Processing CreateTablet for tablet 90efce1de3cc44b9a9640ae62b4f5ee0 (DEFAULT_TABLE table=test-table [id=be9002c64d8f4417b163b5ab7f466677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:45.332893 23460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90efce1de3cc44b9a9640ae62b4f5ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:45.333035 23594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90efce1de3cc44b9a9640ae62b4f5ee0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:45.343036 23810 tablet_bootstrap.cc:492] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Bootstrap starting.
I20260927 23:08:45.346405 23810 tablet_bootstrap.cc:654] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:45.346874 23812 tablet_bootstrap.cc:492] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Bootstrap starting.
I20260927 23:08:45.347842 23810 log.cc:824] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:45.348519 23811 tablet_bootstrap.cc:492] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Bootstrap starting.
I20260927 23:08:45.350525 23812 tablet_bootstrap.cc:654] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:45.351325 23811 tablet_bootstrap.cc:654] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:45.351899 23812 log.cc:824] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:45.352278 23811 log.cc:824] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:45.355176 23812 tablet_bootstrap.cc:492] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: No bootstrap required, opened a new log
I20260927 23:08:45.355444 23812 ts_tablet_manager.cc:1397] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260927 23:08:45.356673 23811 tablet_bootstrap.cc:492] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: No bootstrap required, opened a new log
I20260927 23:08:45.356916 23810 tablet_bootstrap.cc:492] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: No bootstrap required, opened a new log
I20260927 23:08:45.356920 23811 ts_tablet_manager.cc:1397] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260927 23:08:45.357164 23810 ts_tablet_manager.cc:1397] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260927 23:08:45.362242 23812 raft_consensus.cc:348] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.362599 23812 raft_consensus.cc:374] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:45.362739 23812 raft_consensus.cc:729] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eaee110076e452488804377de090a39, State: Initialized, Role: FOLLOWER
I20260927 23:08:45.363355 23812 consensus_queue.cc:260] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.365630 23659 heartbeater.cc:503] Master 127.3.240.254:41151 was elected leader, sending a full tablet report...
I20260927 23:08:45.367347 23810 raft_consensus.cc:348] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.367345 23811 raft_consensus.cc:348] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.367758 23812 ts_tablet_manager.cc:1428] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260927 23:08:45.367753 23811 raft_consensus.cc:374] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:45.367753 23810 raft_consensus.cc:374] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:45.367854 23810 raft_consensus.cc:729] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40d155ca09f48638bf756c49c3c3af8, State: Initialized, Role: FOLLOWER
I20260927 23:08:45.367856 23811 raft_consensus.cc:729] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4da5e595d4246de9a7fea446c0d1b7c, State: Initialized, Role: FOLLOWER
I20260927 23:08:45.368474 23810 consensus_queue.cc:260] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.368467 23811 consensus_queue.cc:260] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.370201 23810 ts_tablet_manager.cc:1428] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
I20260927 23:08:45.370464 23811 ts_tablet_manager.cc:1428] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Time spent starting tablet: real 0.013s user 0.006s sys 0.008s
I20260927 23:08:45.370556 23792 heartbeater.cc:503] Master 127.3.240.254:41151 was elected leader, sending a full tablet report...
W20260927 23:08:45.495404 23793 tablet.cc:2369] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:08:45.498625 23660 tablet.cc:2369] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:08:45.516111 23526 tablet.cc:2369] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:45.553241 23817 raft_consensus.cc:482] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:08:45.553529 23817 raft_consensus.cc:504] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.555269 23817 leader_election.cc:288] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1eaee110076e452488804377de090a39 (127.3.240.194:45955), d4da5e595d4246de9a7fea446c0d1b7c (127.3.240.195:34315)
I20260927 23:08:45.562063 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.555703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:42137 (local address 127.3.240.194:45955)
0927 23:08:45.555912 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.555920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.556726 (+ 806us) server_negotiation.cc:407] Connection header received
0927 23:08:45.556904 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.556910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.557010 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.557093 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:45.557759 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.558504 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.560310 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.561085 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.561298 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.561388 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.561655 (+ 267us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.561788 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260927 23:08:45.562067 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.555703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:33293 (local address 127.3.240.195:34315)
0927 23:08:45.555901 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.555907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.556899 (+ 992us) server_negotiation.cc:407] Connection header received
0927 23:08:45.557039 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.557043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.557130 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.557221 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:45.557866 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.558633 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.560397 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.561100 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.561326 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.561393 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.561727 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.561852 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260927 23:08:45.562278 23819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.555690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:45955 (local address 127.3.240.193:42137)
0927 23:08:45.556591 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:45.556617 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:45.556766 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:45.557131 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:45.557136 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:45.557169 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:45.557607 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.557616 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.558642 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.558646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.560161 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.560172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.561343 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.561349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.561506 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.561529 (+ 23us) client_negotiation.cc:769] Sending connection context
0927 23:08:45.561600 (+ 71us) client_negotiation.cc:240] Negotiation successful
0927 23:08:45.561726 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":87,"thread_start_us":400,"threads_started":1}
I20260927 23:08:45.562465 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.555916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34315 (local address 127.3.240.193:33293)
0927 23:08:45.556754 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:45.556775 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:45.556918 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:45.557258 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:45.557262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:45.557292 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:45.557697 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.557710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.558766 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.558769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.560249 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:45.560267 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.561387 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.561393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:45.561549 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.561571 (+ 22us) client_negotiation.cc:769] Sending connection context
0927 23:08:45.561683 (+ 112us) client_negotiation.cc:240] Negotiation successful
0927 23:08:45.561796 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":2,"thread_start_us":520,"threads_started":1}
I20260927 23:08:45.563341 23747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90efce1de3cc44b9a9640ae62b4f5ee0" candidate_uuid: "c40d155ca09f48638bf756c49c3c3af8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" is_pre_election: true
I20260927 23:08:45.563334 23614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90efce1de3cc44b9a9640ae62b4f5ee0" candidate_uuid: "c40d155ca09f48638bf756c49c3c3af8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaee110076e452488804377de090a39" is_pre_election: true
I20260927 23:08:45.563792 23747 raft_consensus.cc:2454] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40d155ca09f48638bf756c49c3c3af8 in term 0.
I20260927 23:08:45.563781 23614 raft_consensus.cc:2454] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40d155ca09f48638bf756c49c3c3af8 in term 0.
I20260927 23:08:45.564373 23413 leader_election.cc:302] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c40d155ca09f48638bf756c49c3c3af8, d4da5e595d4246de9a7fea446c0d1b7c; no voters:
I20260927 23:08:45.564810 23817 raft_consensus.cc:2781] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:08:45.564936 23817 raft_consensus.cc:482] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:08:45.565052 23817 raft_consensus.cc:3037] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:45.567302 23817 raft_consensus.cc:504] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.567968 23817 leader_election.cc:288] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [CANDIDATE]: Term 1 election: Requested vote from peers 1eaee110076e452488804377de090a39 (127.3.240.194:45955), d4da5e595d4246de9a7fea446c0d1b7c (127.3.240.195:34315)
I20260927 23:08:45.568382 23614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90efce1de3cc44b9a9640ae62b4f5ee0" candidate_uuid: "c40d155ca09f48638bf756c49c3c3af8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eaee110076e452488804377de090a39"
I20260927 23:08:45.568517 23747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90efce1de3cc44b9a9640ae62b4f5ee0" candidate_uuid: "c40d155ca09f48638bf756c49c3c3af8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4da5e595d4246de9a7fea446c0d1b7c"
I20260927 23:08:45.568565 23614 raft_consensus.cc:3037] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:45.568723 23747 raft_consensus.cc:3037] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:45.570704 23614 raft_consensus.cc:2454] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40d155ca09f48638bf756c49c3c3af8 in term 1.
I20260927 23:08:45.570931 23747 raft_consensus.cc:2454] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40d155ca09f48638bf756c49c3c3af8 in term 1.
I20260927 23:08:45.571152 23415 leader_election.cc:302] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1eaee110076e452488804377de090a39, c40d155ca09f48638bf756c49c3c3af8; no voters:
I20260927 23:08:45.571424 23817 raft_consensus.cc:2781] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:45.571885 23817 raft_consensus.cc:686] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 1 LEADER]: Becoming Leader. State: Replica: c40d155ca09f48638bf756c49c3c3af8, State: Running, Role: LEADER
I20260927 23:08:45.572384 23817 consensus_queue.cc:237] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } }
I20260927 23:08:45.576753 23336 catalog_manager.cc:5495] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c40d155ca09f48638bf756c49c3c3af8 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "c40d155ca09f48638bf756c49c3c3af8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c40d155ca09f48638bf756c49c3c3af8" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260927 23:08:45.618216 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.613378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.3.240.193:46407)
0927 23:08:45.613606 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.613611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.613628 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:08:45.613714 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.613717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.613800 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.613938 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:08:45.614325 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.615074 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.616324 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.616711 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.616801 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.617208 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:08:45.617690 (+ 482us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:08:45.617789 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.617867 (+ 78us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.617924 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 23:08:45.626236 23747 raft_consensus.cc:1261] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 1 FOLLOWER]: Refusing update from remote peer c40d155ca09f48638bf756c49c3c3af8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:45.626281 23614 raft_consensus.cc:1261] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 1 FOLLOWER]: Refusing update from remote peer c40d155ca09f48638bf756c49c3c3af8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:45.627281 23821 consensus_queue.cc:1035] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4da5e595d4246de9a7fea446c0d1b7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:45.627647 23817 consensus_queue.cc:1035] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eaee110076e452488804377de090a39" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:45.640913 23824 mvcc.cc:205] Tried to move back new op lower bound from 7334094952954310656 to 7334094952748007424. Current Snapshot: MvccSnapshot[applied={T|T < 7334094952954310656}]
I20260927 23:08:45.662848 23336 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:45.663197 23336 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60786:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:45.666949 23336 catalog_manager.cc:5782] T 00000000000000000000000000000000 P afd20723637846cca53e049e13f37572: Sending DeleteTablet for 3 replicas of tablet 90efce1de3cc44b9a9640ae62b4f5ee0
I20260927 23:08:45.668064 23594 tablet_service.cc:1514] Processing DeleteTablet for tablet 90efce1de3cc44b9a9640ae62b4f5ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53250
I20260927 23:08:45.668128 23727 tablet_service.cc:1514] Processing DeleteTablet for tablet 90efce1de3cc44b9a9640ae62b4f5ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39020
I20260927 23:08:45.668267 23460 tablet_service.cc:1514] Processing DeleteTablet for tablet 90efce1de3cc44b9a9640ae62b4f5ee0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:55178
I20260927 23:08:45.670279 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 23397
I20260927 23:08:45.679502 23835 tablet_replica.cc:330] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: stopping tablet replica
I20260927 23:08:45.680080 23835 raft_consensus.cc:2229] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:45.680802 23835 raft_consensus.cc:2258] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:45.681707 23834 tablet_replica.cc:330] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: stopping tablet replica
I20260927 23:08:45.682610 23836 tablet_replica.cc:330] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: stopping tablet replica
I20260927 23:08:45.683087 23836 raft_consensus.cc:2229] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:08:45.683737 23836 raft_consensus.cc:2258] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:45.684366 23835 ts_tablet_manager.cc:1905] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:45.684407 23834 raft_consensus.cc:2229] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:45.684839 23834 raft_consensus.cc:2258] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:45.686702 23834 ts_tablet_manager.cc:1905] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:45.689411 23836 ts_tablet_manager.cc:1905] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:45.691660 23835 ts_tablet_manager.cc:1918] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:45.691865 23835 log.cc:1196] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/90efce1de3cc44b9a9640ae62b4f5ee0
I20260927 23:08:45.692729 23835 ts_tablet_manager.cc:1939] T 90efce1de3cc44b9a9640ae62b4f5ee0 P 1eaee110076e452488804377de090a39: Deleting consensus metadata
I20260927 23:08:45.693440 23834 ts_tablet_manager.cc:1918] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:45.693593 23834 log.cc:1196] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/90efce1de3cc44b9a9640ae62b4f5ee0
I20260927 23:08:45.694073 23834 ts_tablet_manager.cc:1939] T 90efce1de3cc44b9a9640ae62b4f5ee0 P d4da5e595d4246de9a7fea446c0d1b7c: Deleting consensus metadata
I20260927 23:08:45.694898 23322 catalog_manager.cc:4841] TS 1eaee110076e452488804377de090a39 (127.3.240.194:45955): tablet 90efce1de3cc44b9a9640ae62b4f5ee0 (table test-table [id=be9002c64d8f4417b163b5ab7f466677]) successfully deleted
I20260927 23:08:45.695693 23320 catalog_manager.cc:4841] TS d4da5e595d4246de9a7fea446c0d1b7c (127.3.240.195:34315): tablet 90efce1de3cc44b9a9640ae62b4f5ee0 (table test-table [id=be9002c64d8f4417b163b5ab7f466677]) successfully deleted
Sep 27 23:08:45 dist-test-slave-lmfp krb5kdc[23291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260927 23:08:45.701426 23836 ts_tablet_manager.cc:1918] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:45.701606 23836 log.cc:1196] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790550224434615-4035-0/minicluster-data/ts-0/wal/wals/90efce1de3cc44b9a9640ae62b4f5ee0
I20260927 23:08:45.702132 23836 ts_tablet_manager.cc:1939] T 90efce1de3cc44b9a9640ae62b4f5ee0 P c40d155ca09f48638bf756c49c3c3af8: Deleting consensus metadata
I20260927 23:08:45.703655 23321 catalog_manager.cc:4841] TS c40d155ca09f48638bf756c49c3c3af8 (127.3.240.193:46407): tablet 90efce1de3cc44b9a9640ae62b4f5ee0 (table test-table [id=be9002c64d8f4417b163b5ab7f466677]) successfully deleted
I20260927 23:08:45.705848 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.671912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.3.240.193:46407)
0927 23:08:45.672105 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.672116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.681944 (+ 9828us) server_negotiation.cc:407] Connection header received
0927 23:08:45.682152 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.682158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.682257 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.682448 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:45.684792 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.685963 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.691294 (+ 5331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.692073 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.692187 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.698153 (+ 5966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:45.698168 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:45.698185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:45.698240 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:45.702876 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.703751 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.703758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.703760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.703944 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.704360 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.704365 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.704369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.704701 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:45.705031 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.705262 (+ 231us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.705511 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:08:45.706704 23520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:45.805323 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 23397
I20260927 23:08:45.823935 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 23531
Sep 27 23:08:45 dist-test-slave-lmfp krb5kdc[23291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260927 23:08:45.839248 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.824745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.3.240.194:45955)
0927 23:08:45.824929 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.824935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.824955 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:08:45.825146 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.825156 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.825223 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.825318 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:45.826907 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.827685 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.828952 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.829486 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.829620 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.833849 (+ 4229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:45.833862 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:45.833879 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:45.833915 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:45.836894 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.837609 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.837616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.837619 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.837734 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.838009 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.838012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.838014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.838383 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:45.838601 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.838774 (+ 173us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.838930 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 23:08:45.839929 23654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:45.929796 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 23531
I20260927 23:08:45.946655 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 23664
Sep 27 23:08:45 dist-test-slave-lmfp krb5kdc[23291](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260927 23:08:45.961022 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:45.947454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.3.240.195:34315)
0927 23:08:45.947657 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:45.947662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:45.947680 (+ 18us) server_negotiation.cc:407] Connection header received
0927 23:08:45.947792 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:45.947815 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:45.947879 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:45.947960 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:45.949206 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.949916 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.951187 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:45.951564 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:45.951656 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:45.955995 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:45.956009 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:45.956025 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:45.956054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:45.958724 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.959381 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.959388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.959390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.959524 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:45.959827 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:45.959830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:45.959832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:45.960140 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:45.960347 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:45.960515 (+ 168us) server_negotiation.cc:299] Negotiation successful
0927 23:08:45.960660 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260927 23:08:45.961619 23787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:46.049191 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 23664
I20260927 23:08:46.066103 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 23304
I20260927 23:08:46.067356 23366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:46.189669 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 23304
2026-09-27T23:08:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4241 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-27T23:08:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:46Z Disabled control of system clock
I20260927 23:08:46.268402 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37135
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 23:08:46.593432 23845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:46.593825 23845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:46.594000 23845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:46.601594 23845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:46.601744 23845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:46.601812 23845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:46.601905 23845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:46.601972 23845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:46.611614 23845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:46.612833 23845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:46.614558 23845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:46.627225 23850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:46.627277 23851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:46.627279 23853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:46.628225 23845 server_base.cc:1034] running on GCE node
I20260927 23:08:46.629067 23845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:46.630456 23845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:46.631650 23845 hybrid_clock.cc:648] HybridClock initialized: now 1790550526631590 us; error 50 us; skew 500 ppm
I20260927 23:08:46.632009 23845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:46.634439 23845 webserver.cc:460] Webserver started at http://127.3.240.254:39729/ using document root <none> and password file <none>
I20260927 23:08:46.635211 23845 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:46.635327 23845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:46.635676 23845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:46.639209 23845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "2a49decdd1934f168cdebefe7dd3ffb2"
format_stamp: "Formatted at 2026-09-27 23:08:46 on dist-test-slave-lmfp"
server_key: "9341be5143734ef98ae34c974328f22b"
server_key_iv: "d912d2c64e236dd3577bf608eecbeaf4"
server_key_version: "encryptionkey@0"
I20260927 23:08:46.640275 23845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2a49decdd1934f168cdebefe7dd3ffb2"
format_stamp: "Formatted at 2026-09-27 23:08:46 on dist-test-slave-lmfp"
server_key: "9341be5143734ef98ae34c974328f22b"
server_key_iv: "d912d2c64e236dd3577bf608eecbeaf4"
server_key_version: "encryptionkey@0"
I20260927 23:08:46.645560 23845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:08:46.649118 23859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:46.650480 23845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:08:46.650686 23845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "2a49decdd1934f168cdebefe7dd3ffb2"
format_stamp: "Formatted at 2026-09-27 23:08:46 on dist-test-slave-lmfp"
server_key: "9341be5143734ef98ae34c974328f22b"
server_key_iv: "d912d2c64e236dd3577bf608eecbeaf4"
server_key_version: "encryptionkey@0"
I20260927 23:08:46.650911 23845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:46.668711 23845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:46.669512 23845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:46.669802 23845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:46.688045 23845 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:37135
I20260927 23:08:46.688055 23910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:37135 every 8 connection(s)
I20260927 23:08:46.689389 23845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:46.690135 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 23845
I20260927 23:08:46.690317 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:46.690908 4035 external_mini_cluster.cc:1428] Setting key b96b947b695964d3a0c966bd6902d801
I20260927 23:08:46.694046 23911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:46.706513 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:46.692855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.3.240.254:37135)
0927 23:08:46.693923 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:46.693937 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:46.693990 (+ 53us) server_negotiation.cc:407] Connection header received
0927 23:08:46.696337 (+ 2347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:46.696365 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:46.696941 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:46.697394 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:46.697947 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:46.700015 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:46.701318 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:46.702333 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:46.702520 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:46.703627 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:46.703639 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:46.703663 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:46.704426 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:46.704519 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:46.704570 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:46.704692 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:46.704894 (+ 202us) server_negotiation.cc:299] Negotiation successful
0927 23:08:46.705241 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":341,"threads_started":1}
I20260927 23:08:46.708177 23911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2: Bootstrap starting.
I20260927 23:08:46.711709 23911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:46.712700 23911 log.cc:824] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:46.715653 23911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2: No bootstrap required, opened a new log
I20260927 23:08:46.722885 23911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a49decdd1934f168cdebefe7dd3ffb2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37135 } }
I20260927 23:08:46.723265 23911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:46.723342 23911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a49decdd1934f168cdebefe7dd3ffb2, State: Initialized, Role: FOLLOWER
I20260927 23:08:46.723806 23911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a49decdd1934f168cdebefe7dd3ffb2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37135 } }
I20260927 23:08:46.724010 23911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:46.724112 23911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:46.724255 23911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:46.726774 23911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a49decdd1934f168cdebefe7dd3ffb2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37135 } }
I20260927 23:08:46.727296 23911 leader_election.cc:302] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a49decdd1934f168cdebefe7dd3ffb2; no voters:
I20260927 23:08:46.727849 23911 leader_election.cc:288] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:46.728016 23916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:46.728720 23916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [term 1 LEADER]: Becoming Leader. State: Replica: 2a49decdd1934f168cdebefe7dd3ffb2, State: Running, Role: LEADER
I20260927 23:08:46.729199 23916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a49decdd1934f168cdebefe7dd3ffb2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37135 } }
I20260927 23:08:46.729887 23911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:46.732288 23918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a49decdd1934f168cdebefe7dd3ffb2. Latest consensus state: current_term: 1 leader_uuid: "2a49decdd1934f168cdebefe7dd3ffb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a49decdd1934f168cdebefe7dd3ffb2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37135 } } }
I20260927 23:08:46.732365 23917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a49decdd1934f168cdebefe7dd3ffb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a49decdd1934f168cdebefe7dd3ffb2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37135 } } }
I20260927 23:08:46.732658 23918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:46.732785 23917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:46.735131 23923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:46.740672 23923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:46.749347 23923 catalog_manager.cc:1347] Generated new cluster ID: 1e04295278ec495db563209f1ef1a01e
I20260927 23:08:46.749475 23923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:46.767108 23923 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:46.768499 23923 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:46.777359 23923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2: Generated new TSK 0
I20260927 23:08:46.778167 23923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:46.783260 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37135
--builtin_ntp_servers=127.3.240.212:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260927 23:08:47.103318 23935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:47.103721 23935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:47.103863 23935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:47.112977 23935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:47.113091 23935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:47.113248 23935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260927 23:08:47.122283 23935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:47.123605 23935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:47.125270 23935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:47.138173 23940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:47.138288 23941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:47.138296 23943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:47.139029 23935 server_base.cc:1034] running on GCE node
I20260927 23:08:47.139655 23935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:47.140496 23935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:47.141723 23935 hybrid_clock.cc:648] HybridClock initialized: now 1790550527141689 us; error 44 us; skew 500 ppm
I20260927 23:08:47.142026 23935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:47.144196 23935 webserver.cc:460] Webserver started at http://127.3.240.193:33713/ using document root <none> and password file <none>
I20260927 23:08:47.145084 23935 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:47.145191 23935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:47.145541 23935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:47.148227 23935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/data/instance:
uuid: "a04558c4f9ff42caae0a40529d34927f"
format_stamp: "Formatted at 2026-09-27 23:08:47 on dist-test-slave-lmfp"
server_key: "e43cf17abd859a50eb047c5285054c3a"
server_key_iv: "48b7264f7325dfc0cf8a3df5fc58b13f"
server_key_version: "encryptionkey@0"
I20260927 23:08:47.148990 23935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "a04558c4f9ff42caae0a40529d34927f"
format_stamp: "Formatted at 2026-09-27 23:08:47 on dist-test-slave-lmfp"
server_key: "e43cf17abd859a50eb047c5285054c3a"
server_key_iv: "48b7264f7325dfc0cf8a3df5fc58b13f"
server_key_version: "encryptionkey@0"
I20260927 23:08:47.154110 23935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 23:08:47.157894 23949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:47.159294 23935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260927 23:08:47.159507 23935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/wal
uuid: "a04558c4f9ff42caae0a40529d34927f"
format_stamp: "Formatted at 2026-09-27 23:08:47 on dist-test-slave-lmfp"
server_key: "e43cf17abd859a50eb047c5285054c3a"
server_key_iv: "48b7264f7325dfc0cf8a3df5fc58b13f"
server_key_version: "encryptionkey@0"
I20260927 23:08:47.159718 23935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:47.190474 23935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:47.191358 23935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:47.191659 23935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:47.192579 23935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:47.193969 23935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:47.194092 23935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:47.194219 23935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:47.194305 23935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:47.227114 23935 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:33237
I20260927 23:08:47.227218 24061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:33237 every 8 connection(s)
I20260927 23:08:47.228415 23935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/data/info.pb
I20260927 23:08:47.229667 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 23935
I20260927 23:08:47.229871 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-0/wal/instance
I20260927 23:08:47.230374 4035 external_mini_cluster.cc:1428] Setting key ce16db5097afb07ac12e5678af2f6610
I20260927 23:08:47.237476 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37135
--builtin_ntp_servers=127.3.240.212:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 23:08:47.243125 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:47.231315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:50803 (local address 127.3.240.254:37135)
0927 23:08:47.235325 (+ 4010us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:47.235331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:47.235346 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:08:47.235393 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:47.235399 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:47.235487 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:47.235574 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:47.236864 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:47.237903 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:47.240327 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:47.240714 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:47.240805 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:47.242306 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:47.242310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:47.242313 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:47.242328 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:47.242363 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:47.242373 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:47.242444 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:47.242744 (+ 300us) server_negotiation.cc:299] Negotiation successful
0927 23:08:47.242815 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3898,"thread_start_us":3815,"threads_started":1}
I20260927 23:08:47.244421 24065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:47.231840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37135 (local address 127.3.240.193:50803)
0927 23:08:47.232803 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:47.232842 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:47.234578 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:47.235871 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:47.235881 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:47.235983 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:47.236696 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:47.236718 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:47.238055 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:47.238063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:47.240188 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:47.240205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:47.240927 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:47.240935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:47.241253 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:47.241978 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:47.241983 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:47.242030 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:47.242124 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:47.242473 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:47.242487 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:47.242520 (+ 33us) client_negotiation.cc:769] Sending connection context
0927 23:08:47.242699 (+ 179us) client_negotiation.cc:240] Negotiation successful
0927 23:08:47.243120 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":318,"threads_started":1}
I20260927 23:08:47.245810 24062 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37135
I20260927 23:08:47.246073 24062 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:47.246884 24062 heartbeater.cc:511] Master 127.3.240.254:37135 requested a full tablet report, sending...
I20260927 23:08:47.248858 23876 ts_manager.cc:194] Registered new tserver with Master: a04558c4f9ff42caae0a40529d34927f (127.3.240.193:33237)
I20260927 23:08:47.250207 23876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.193:50803
W20260927 23:08:47.544175 24066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:47.544415 24066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:47.544553 24066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:47.551180 24066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:47.551257 24066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:47.551435 24066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260927 23:08:47.559885 24066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:47.561146 24066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:47.562856 24066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:47.575536 24072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:47.575488 24074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:47.575567 24071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:47.576268 24066 server_base.cc:1034] running on GCE node
I20260927 23:08:47.576864 24066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:47.577759 24066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:47.578981 24066 hybrid_clock.cc:648] HybridClock initialized: now 1790550527578939 us; error 53 us; skew 500 ppm
I20260927 23:08:47.579344 24066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:47.581377 24066 webserver.cc:460] Webserver started at http://127.3.240.194:37123/ using document root <none> and password file <none>
I20260927 23:08:47.582044 24066 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:47.582163 24066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:47.582515 24066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:47.585326 24066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/data/instance:
uuid: "84cfdfda5501416ea8c239494f0ffe56"
format_stamp: "Formatted at 2026-09-27 23:08:47 on dist-test-slave-lmfp"
server_key: "88b45672bd139d2d98cbe1c9edfa4947"
server_key_iv: "adc8d60aa0914a835496b5a81d50867b"
server_key_version: "encryptionkey@0"
I20260927 23:08:47.586117 24066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "84cfdfda5501416ea8c239494f0ffe56"
format_stamp: "Formatted at 2026-09-27 23:08:47 on dist-test-slave-lmfp"
server_key: "88b45672bd139d2d98cbe1c9edfa4947"
server_key_iv: "adc8d60aa0914a835496b5a81d50867b"
server_key_version: "encryptionkey@0"
I20260927 23:08:47.591048 24066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 23:08:47.594511 24080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:47.595860 24066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:08:47.596047 24066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/wal
uuid: "84cfdfda5501416ea8c239494f0ffe56"
format_stamp: "Formatted at 2026-09-27 23:08:47 on dist-test-slave-lmfp"
server_key: "88b45672bd139d2d98cbe1c9edfa4947"
server_key_iv: "adc8d60aa0914a835496b5a81d50867b"
server_key_version: "encryptionkey@0"
I20260927 23:08:47.596267 24066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:47.610447 24066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:47.611279 24066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:47.611563 24066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:47.612465 24066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:47.613799 24066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:47.613914 24066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:47.614071 24066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:47.614163 24066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:47.647329 24066 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:43971
I20260927 23:08:47.647374 24192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:43971 every 8 connection(s)
I20260927 23:08:47.648586 24066 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/data/info.pb
I20260927 23:08:47.652654 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 24066
I20260927 23:08:47.652818 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/wal/instance
I20260927 23:08:47.653348 4035 external_mini_cluster.cc:1428] Setting key a29e7c589739b707b2e1cbe3c7d0636d
I20260927 23:08:47.656284 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37135
--builtin_ntp_servers=127.3.240.212:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260927 23:08:47.661966 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:47.651077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:54431 (local address 127.3.240.254:37135)
0927 23:08:47.651249 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:47.651253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:47.652601 (+ 1348us) server_negotiation.cc:407] Connection header received
0927 23:08:47.654905 (+ 2304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:47.654909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:47.654971 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:47.655130 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:47.656213 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:47.657360 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:47.658943 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:47.659671 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:47.659824 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:47.661053 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:47.661062 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:47.661069 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:47.661094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:47.661137 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:47.661157 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:47.661282 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:47.661473 (+ 191us) server_negotiation.cc:299] Negotiation successful
0927 23:08:47.661577 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:08:47.662787 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:47.651510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37135 (local address 127.3.240.194:54431)
0927 23:08:47.652406 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:47.652464 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:47.654636 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:47.655380 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:47.655390 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:47.655496 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:47.656048 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:47.656064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:47.657530 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:47.657534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:47.658796 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:47.658807 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:47.659869 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:47.659874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:47.660070 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:47.660749 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:47.660751 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:47.660787 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:47.660856 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:47.661282 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:47.661289 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:47.661294 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 23:08:47.661424 (+ 130us) client_negotiation.cc:240] Negotiation successful
0927 23:08:47.661702 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":253,"threads_started":1}
I20260927 23:08:47.664368 24193 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37135
I20260927 23:08:47.664693 24193 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:47.665623 24193 heartbeater.cc:511] Master 127.3.240.254:37135 requested a full tablet report, sending...
I20260927 23:08:47.667048 23876 ts_manager.cc:194] Registered new tserver with Master: 84cfdfda5501416ea8c239494f0ffe56 (127.3.240.194:43971)
I20260927 23:08:47.667845 23876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.194:54431
W20260927 23:08:47.964856 24196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:47.965092 24196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:47.965230 24196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:47.972078 24196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:47.972168 24196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:47.972340 24196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260927 23:08:47.980749 24196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37135
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:47.982053 24196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:47.983841 24196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:47.996942 24201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:47.996858 24204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:47.996856 24202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:47.997684 24196 server_base.cc:1034] running on GCE node
I20260927 23:08:47.998247 24196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:47.999290 24196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:48.000514 24196 hybrid_clock.cc:648] HybridClock initialized: now 1790550528000465 us; error 38 us; skew 500 ppm
I20260927 23:08:48.000777 24196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260927 23:08:48.002900 24196 webserver.cc:460] Webserver started at http://127.3.240.195:42065/ using document root <none> and password file <none>
I20260927 23:08:48.003597 24196 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:48.003711 24196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:48.004031 24196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:48.006732 24196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/data/instance:
uuid: "f4ac24cb73ac46af94843a702b8651d1"
format_stamp: "Formatted at 2026-09-27 23:08:48 on dist-test-slave-lmfp"
server_key: "11909177d80b80eef3b8dede789356fd"
server_key_iv: "52b93c9ca7c14019da2f7467b2bf00aa"
server_key_version: "encryptionkey@0"
I20260927 23:08:48.007578 24196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "f4ac24cb73ac46af94843a702b8651d1"
format_stamp: "Formatted at 2026-09-27 23:08:48 on dist-test-slave-lmfp"
server_key: "11909177d80b80eef3b8dede789356fd"
server_key_iv: "52b93c9ca7c14019da2f7467b2bf00aa"
server_key_version: "encryptionkey@0"
I20260927 23:08:48.012187 24196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:08:48.015439 24210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:48.016803 24196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260927 23:08:48.016978 24196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/wal
uuid: "f4ac24cb73ac46af94843a702b8651d1"
format_stamp: "Formatted at 2026-09-27 23:08:48 on dist-test-slave-lmfp"
server_key: "11909177d80b80eef3b8dede789356fd"
server_key_iv: "52b93c9ca7c14019da2f7467b2bf00aa"
server_key_version: "encryptionkey@0"
I20260927 23:08:48.017204 24196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:48.042555 24196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:48.043442 24196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:48.043746 24196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:48.044691 24196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260927 23:08:48.046095 24196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260927 23:08:48.046177 24196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:48.046327 24196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260927 23:08:48.046414 24196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:48.079995 24196 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42441
I20260927 23:08:48.080052 24322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42441 every 8 connection(s)
I20260927 23:08:48.081375 24196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/data/info.pb
I20260927 23:08:48.082636 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 24196
I20260927 23:08:48.082791 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/wal/instance
I20260927 23:08:48.083319 4035 external_mini_cluster.cc:1428] Setting key 3bbabb5df221aac4d992f4f452b97cd7
I20260927 23:08:48.093771 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.084138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:41117 (local address 127.3.240.254:37135)
0927 23:08:48.084378 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.084385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.086174 (+ 1789us) server_negotiation.cc:407] Connection header received
0927 23:08:48.088004 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.088008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.088074 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.088176 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:48.089185 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.089846 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.091440 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.091836 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.091927 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.093153 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:48.093158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:48.093161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:48.093176 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:48.093206 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:48.093218 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:48.093286 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.093479 (+ 193us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.093552 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260927 23:08:48.094750 24325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.084864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37135 (local address 127.3.240.195:41117)
0927 23:08:48.086017 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.086068 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:48.087774 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:48.088403 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:48.088410 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:48.088495 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:08:48.089040 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.089056 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.089978 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.089981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.091303 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.091316 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.092025 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.092029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.092228 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.092867 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0927 23:08:48.092869 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:08:48.092910 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0927 23:08:48.092989 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:08:48.093302 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:08:48.093310 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:08:48.093315 (+ 5us) client_negotiation.cc:769] Sending connection context
0927 23:08:48.093462 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 23:08:48.093807 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":387,"threads_started":1}
I20260927 23:08:48.095952 24323 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37135
I20260927 23:08:48.096266 24323 heartbeater.cc:464] Registering TS with master...
I20260927 23:08:48.097074 24323 heartbeater.cc:511] Master 127.3.240.254:37135 requested a full tablet report, sending...
I20260927 23:08:48.098489 23876 ts_manager.cc:194] Registered new tserver with Master: f4ac24cb73ac46af94843a702b8651d1 (127.3.240.195:42441)
I20260927 23:08:48.099471 23876 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.195:41117
I20260927 23:08:48.109742 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260927 23:08:48.118906 24064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.113507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42676 (local address 127.3.240.254:37135)
0927 23:08:48.113660 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.113664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.114166 (+ 502us) server_negotiation.cc:407] Connection header received
0927 23:08:48.114286 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.114290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.114343 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.114415 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:48.114877 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.115624 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.116954 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.117319 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.117404 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.118372 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:48.118377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:48.118380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:48.118395 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:48.118424 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:48.118435 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:48.118502 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.118633 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.118700 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:08:48.122357 23876 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260927 23:08:48.128961 23876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260927 23:08:48.163029 24340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.140409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:43971 (local address 127.0.0.1:33070)
0927 23:08:48.151419 (+ 11010us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.151483 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:48.152717 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:48.154316 (+ 1599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:48.154326 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:48.154374 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:48.154838 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.154854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.158247 (+ 3393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.158252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.160675 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.160686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.161988 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.161993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.162219 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.162300 (+ 81us) client_negotiation.cc:769] Sending connection context
0927 23:08:48.162428 (+ 128us) client_negotiation.cc:240] Negotiation successful
0927 23:08:48.162618 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10647,"mutex_wait_us":1,"thread_start_us":16454,"threads_started":1}
I20260927 23:08:48.163751 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.140731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33070 (local address 127.3.240.194:43971)
0927 23:08:48.141492 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.141505 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.151660 (+ 10155us) server_negotiation.cc:407] Connection header received
0927 23:08:48.153332 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.153344 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.153665 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.153855 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:48.156551 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.158094 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.160870 (+ 2776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.161794 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.161952 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.162271 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.162489 (+ 218us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.162644 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":853,"threads_started":1}
I20260927 23:08:48.165205 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.142352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.0.0.1:42688)
0927 23:08:48.156597 (+ 14245us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.156620 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:48.156760 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:48.157677 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:48.157681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:48.157705 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:48.158231 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.158248 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.160464 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.160469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.162715 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.162729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.164617 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.164624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.164788 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.164824 (+ 36us) client_negotiation.cc:769] Sending connection context
0927 23:08:48.164911 (+ 87us) client_negotiation.cc:240] Negotiation successful
0927 23:08:48.164989 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14079,"mutex_wait_us":74,"thread_start_us":8507,"threads_started":1}
I20260927 23:08:48.165647 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.141981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42688 (local address 127.3.240.193:33237)
0927 23:08:48.157112 (+ 15131us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.157121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.157149 (+ 28us) server_negotiation.cc:407] Connection header received
0927 23:08:48.157244 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.157251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.157463 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.157668 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:48.158460 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.160334 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.162920 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.164407 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.164532 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.164743 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.164984 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.165146 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14949,"thread_start_us":14997,"threads_started":1}
I20260927 23:08:48.167522 24128 tablet_service.cc:1467] Processing CreateTablet for tablet 6df95671d9dd4c23805596bd1ae50ca6 (DEFAULT_TABLE table=test-table [id=249e5db8670b4cdcb5fe304ce86f1413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:48.168785 24128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df95671d9dd4c23805596bd1ae50ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:48.168615 23997 tablet_service.cc:1467] Processing CreateTablet for tablet 6df95671d9dd4c23805596bd1ae50ca6 (DEFAULT_TABLE table=test-table [id=249e5db8670b4cdcb5fe304ce86f1413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:48.169603 23997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df95671d9dd4c23805596bd1ae50ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:48.174243 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.141876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.3.240.195:42441)
0927 23:08:48.157360 (+ 15484us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.157366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.157390 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:08:48.157462 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.157467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.157686 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.157879 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:48.158769 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.160091 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.163414 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.172522 (+ 9108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.172946 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.173210 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.173719 (+ 509us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.173892 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15310,"thread_start_us":16041,"threads_started":1}
I20260927 23:08:48.174393 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.141421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42441 (local address 127.0.0.1:43840)
0927 23:08:48.155803 (+ 14382us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.155820 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:48.155922 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:48.157892 (+ 1970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:48.157896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:48.157918 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:48.158566 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.158575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.160220 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.160227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.163152 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.163166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.173328 (+ 10162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.173335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.173517 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.173549 (+ 32us) client_negotiation.cc:769] Sending connection context
0927 23:08:48.173647 (+ 98us) client_negotiation.cc:240] Negotiation successful
0927 23:08:48.173781 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14240,"mutex_wait_us":136,"thread_start_us":14069,"threads_started":1}
I20260927 23:08:48.177978 24258 tablet_service.cc:1467] Processing CreateTablet for tablet 6df95671d9dd4c23805596bd1ae50ca6 (DEFAULT_TABLE table=test-table [id=249e5db8670b4cdcb5fe304ce86f1413]), partition=RANGE (key) PARTITION UNBOUNDED
I20260927 23:08:48.179591 24258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6df95671d9dd4c23805596bd1ae50ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:48.180404 24341 tablet_bootstrap.cc:492] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: Bootstrap starting.
I20260927 23:08:48.182224 24342 tablet_bootstrap.cc:492] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: Bootstrap starting.
I20260927 23:08:48.183915 24341 tablet_bootstrap.cc:654] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:48.185161 24341 log.cc:824] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:48.185708 24342 tablet_bootstrap.cc:654] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:48.186887 24342 log.cc:824] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:48.188272 24341 tablet_bootstrap.cc:492] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: No bootstrap required, opened a new log
I20260927 23:08:48.188529 24341 ts_tablet_manager.cc:1397] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260927 23:08:48.189685 24342 tablet_bootstrap.cc:492] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: No bootstrap required, opened a new log
I20260927 23:08:48.189939 24342 ts_tablet_manager.cc:1397] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260927 23:08:48.195729 24345 tablet_bootstrap.cc:492] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: Bootstrap starting.
I20260927 23:08:48.198714 24345 tablet_bootstrap.cc:654] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:48.198943 24341 raft_consensus.cc:348] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.199424 24341 raft_consensus.cc:374] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:48.199527 24341 raft_consensus.cc:729] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84cfdfda5501416ea8c239494f0ffe56, State: Initialized, Role: FOLLOWER
I20260927 23:08:48.199971 24345 log.cc:824] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:48.200145 24341 consensus_queue.cc:260] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.200212 24342 raft_consensus.cc:348] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.200702 24342 raft_consensus.cc:374] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:48.200851 24342 raft_consensus.cc:729] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04558c4f9ff42caae0a40529d34927f, State: Initialized, Role: FOLLOWER
I20260927 23:08:48.201593 24342 consensus_queue.cc:260] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.202891 24062 heartbeater.cc:503] Master 127.3.240.254:37135 was elected leader, sending a full tablet report...
I20260927 23:08:48.203711 24342 ts_tablet_manager.cc:1428] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260927 23:08:48.204955 24193 heartbeater.cc:503] Master 127.3.240.254:37135 was elected leader, sending a full tablet report...
I20260927 23:08:48.206801 24345 tablet_bootstrap.cc:492] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: No bootstrap required, opened a new log
I20260927 23:08:48.207041 24341 ts_tablet_manager.cc:1428] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260927 23:08:48.207126 24345 ts_tablet_manager.cc:1397] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260927 23:08:48.214493 24345 raft_consensus.cc:348] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.214830 24345 raft_consensus.cc:374] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:48.214943 24345 raft_consensus.cc:729] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4ac24cb73ac46af94843a702b8651d1, State: Initialized, Role: FOLLOWER
I20260927 23:08:48.215503 24345 consensus_queue.cc:260] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.216398 24323 heartbeater.cc:503] Master 127.3.240.254:37135 was elected leader, sending a full tablet report...
I20260927 23:08:48.217033 24345 ts_tablet_manager.cc:1428] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
W20260927 23:08:48.230535 24063 tablet.cc:2369] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:08:48.246670 24347 raft_consensus.cc:482] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:08:48.246989 24347 raft_consensus.cc:504] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.248608 24347 leader_election.cc:288] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4ac24cb73ac46af94843a702b8651d1 (127.3.240.195:42441), a04558c4f9ff42caae0a40529d34927f (127.3.240.193:33237)
I20260927 23:08:48.257136 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.248984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:35935 (local address 127.3.240.195:42441)
0927 23:08:48.249195 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.249204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.249697 (+ 493us) server_negotiation.cc:407] Connection header received
0927 23:08:48.249886 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.249889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.249981 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.250085 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:48.250747 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.251863 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.254748 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.255978 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.256131 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.256198 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.256780 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.256897 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:08:48.257153 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.249216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:60249 (local address 127.3.240.193:33237)
0927 23:08:48.249441 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.249449 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.249957 (+ 508us) server_negotiation.cc:407] Connection header received
0927 23:08:48.250062 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.250069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.250147 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.250230 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0927 23:08:48.250864 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.251878 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.254757 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.255975 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.256131 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.256198 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.256780 (+ 582us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.256900 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 23:08:48.257333 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.248830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42441 (local address 127.3.240.194:35935)
0927 23:08:48.249560 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.249585 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:48.249754 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:48.250104 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:48.250107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:48.250124 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:48.250596 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.250614 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.251992 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.251997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.254531 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.254545 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.256200 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.256206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.256529 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.256646 (+ 117us) client_negotiation.cc:769] Sending connection context
0927 23:08:48.256737 (+ 91us) client_negotiation.cc:240] Negotiation successful
0927 23:08:48.256939 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":281,"spinlock_wait_cycles":19328,"thread_start_us":749,"threads_started":1}
I20260927 23:08:48.257378 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.248881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.3.240.194:60249)
0927 23:08:48.249852 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.249868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0927 23:08:48.249955 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:08:48.250263 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:08:48.250268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:08:48.250294 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0927 23:08:48.250710 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.250721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.251994 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.251997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.254533 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:08:48.254545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.256208 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.256213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:08:48.256535 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.256644 (+ 109us) client_negotiation.cc:769] Sending connection context
0927 23:08:48.256737 (+ 93us) client_negotiation.cc:240] Negotiation successful
0927 23:08:48.256939 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"mutex_wait_us":331,"thread_start_us":329,"threads_started":1}
I20260927 23:08:48.258396 24017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df95671d9dd4c23805596bd1ae50ca6" candidate_uuid: "84cfdfda5501416ea8c239494f0ffe56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04558c4f9ff42caae0a40529d34927f" is_pre_election: true
I20260927 23:08:48.258405 24278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df95671d9dd4c23805596bd1ae50ca6" candidate_uuid: "84cfdfda5501416ea8c239494f0ffe56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4ac24cb73ac46af94843a702b8651d1" is_pre_election: true
I20260927 23:08:48.258869 24017 raft_consensus.cc:2454] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cfdfda5501416ea8c239494f0ffe56 in term 0.
I20260927 23:08:48.258883 24278 raft_consensus.cc:2454] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84cfdfda5501416ea8c239494f0ffe56 in term 0.
I20260927 23:08:48.259527 24083 leader_election.cc:302] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 84cfdfda5501416ea8c239494f0ffe56, a04558c4f9ff42caae0a40529d34927f, f4ac24cb73ac46af94843a702b8651d1; no voters:
I20260927 23:08:48.259907 24347 raft_consensus.cc:2781] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260927 23:08:48.260152 24347 raft_consensus.cc:482] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260927 23:08:48.260277 24347 raft_consensus.cc:3037] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:48.262413 24347 raft_consensus.cc:504] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.263039 24347 leader_election.cc:288] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [CANDIDATE]: Term 1 election: Requested vote from peers f4ac24cb73ac46af94843a702b8651d1 (127.3.240.195:42441), a04558c4f9ff42caae0a40529d34927f (127.3.240.193:33237)
I20260927 23:08:48.263475 24278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df95671d9dd4c23805596bd1ae50ca6" candidate_uuid: "84cfdfda5501416ea8c239494f0ffe56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4ac24cb73ac46af94843a702b8651d1"
I20260927 23:08:48.263499 24017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6df95671d9dd4c23805596bd1ae50ca6" candidate_uuid: "84cfdfda5501416ea8c239494f0ffe56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04558c4f9ff42caae0a40529d34927f"
I20260927 23:08:48.263695 24017 raft_consensus.cc:3037] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:48.263691 24278 raft_consensus.cc:3037] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:48.266826 24017 raft_consensus.cc:2454] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cfdfda5501416ea8c239494f0ffe56 in term 1.
I20260927 23:08:48.266826 24278 raft_consensus.cc:2454] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84cfdfda5501416ea8c239494f0ffe56 in term 1.
I20260927 23:08:48.267244 24083 leader_election.cc:302] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84cfdfda5501416ea8c239494f0ffe56, f4ac24cb73ac46af94843a702b8651d1; no voters:
I20260927 23:08:48.267565 24347 raft_consensus.cc:2781] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:48.267987 24347 raft_consensus.cc:686] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 1 LEADER]: Becoming Leader. State: Replica: 84cfdfda5501416ea8c239494f0ffe56, State: Running, Role: LEADER
I20260927 23:08:48.268484 24347 consensus_queue.cc:237] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } }
I20260927 23:08:48.272701 23876 catalog_manager.cc:5495] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84cfdfda5501416ea8c239494f0ffe56 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "84cfdfda5501416ea8c239494f0ffe56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84cfdfda5501416ea8c239494f0ffe56" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 } health_report { overall_health: UNKNOWN } } }
I20260927 23:08:48.288421 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.283255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.3.240.194:43971)
0927 23:08:48.283416 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.283424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.283503 (+ 79us) server_negotiation.cc:407] Connection header received
0927 23:08:48.283743 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.283746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.283834 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.283976 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:08:48.284333 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.285010 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.286362 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.286783 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.286927 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.287433 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:08:48.287884 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:08:48.288037 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.288111 (+ 74us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.288184 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:08:48.295853 24017 raft_consensus.cc:1261] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 1 FOLLOWER]: Refusing update from remote peer 84cfdfda5501416ea8c239494f0ffe56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:48.295858 24278 raft_consensus.cc:1261] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 1 FOLLOWER]: Refusing update from remote peer 84cfdfda5501416ea8c239494f0ffe56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:08:48.296847 24352 consensus_queue.cc:1035] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04558c4f9ff42caae0a40529d34927f" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:48.297183 24347 consensus_queue.cc:1035] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4ac24cb73ac46af94843a702b8651d1" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:08:48.310053 24357 mvcc.cc:205] Tried to move back new op lower bound from 7334094963890552832 to 7334094963791532032. Current Snapshot: MvccSnapshot[applied={T|T < 7334094963890552832}]
I20260927 23:08:48.324889 23875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:48.325166 23875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260927 23:08:48.328533 23875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2a49decdd1934f168cdebefe7dd3ffb2: Sending DeleteTablet for 3 replicas of tablet 6df95671d9dd4c23805596bd1ae50ca6
I20260927 23:08:48.329514 24128 tablet_service.cc:1514] Processing DeleteTablet for tablet 6df95671d9dd4c23805596bd1ae50ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:48 UTC) from {username='slave'} at 127.0.0.1:33070
I20260927 23:08:48.329850 24258 tablet_service.cc:1514] Processing DeleteTablet for tablet 6df95671d9dd4c23805596bd1ae50ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:48 UTC) from {username='slave'} at 127.0.0.1:43840
I20260927 23:08:48.329846 23997 tablet_service.cc:1514] Processing DeleteTablet for tablet 6df95671d9dd4c23805596bd1ae50ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-27 23:08:48 UTC) from {username='slave'} at 127.0.0.1:42688
I20260927 23:08:48.331904 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 23935
I20260927 23:08:48.340408 24365 tablet_replica.cc:330] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: stopping tablet replica
I20260927 23:08:48.340834 24365 raft_consensus.cc:2229] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 1 LEADER]: Raft consensus shutting down.
I20260927 23:08:48.341504 24365 raft_consensus.cc:2258] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:48.343554 24367 tablet_replica.cc:330] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: stopping tablet replica
I20260927 23:08:48.343999 24367 raft_consensus.cc:2229] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:48.344565 24367 raft_consensus.cc:2258] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:48.345666 24365 ts_tablet_manager.cc:1905] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:48.346201 24366 tablet_replica.cc:330] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: stopping tablet replica
I20260927 23:08:48.346526 24366 raft_consensus.cc:2229] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260927 23:08:48.346578 24367 ts_tablet_manager.cc:1905] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:48.346889 24366 raft_consensus.cc:2258] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260927 23:08:48.351094 24338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.333103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.3.240.193:33237)
0927 23:08:48.333353 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.333361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.343509 (+ 10148us) server_negotiation.cc:407] Connection header received
0927 23:08:48.343799 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.343804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.343941 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.344028 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:48.344743 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.345908 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.347525 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.348091 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.348205 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.349472 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:48.349483 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:48.349499 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:48.350206 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:48.350260 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:48.350282 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:48.350363 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.350583 (+ 220us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.350682 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 23:08:48.352317 24057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:48.351118 24366 ts_tablet_manager.cc:1905] T 6df95671d9dd4c23805596bd1ae50ca6 P a04558c4f9ff42caae0a40529d34927f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260927 23:08:48.355621 24365 ts_tablet_manager.cc:1918] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:48.355837 24365 log.cc:1196] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-1/wal/wals/6df95671d9dd4c23805596bd1ae50ca6
I20260927 23:08:48.356171 24367 ts_tablet_manager.cc:1918] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260927 23:08:48.356402 24365 ts_tablet_manager.cc:1939] T 6df95671d9dd4c23805596bd1ae50ca6 P 84cfdfda5501416ea8c239494f0ffe56: Deleting consensus metadata
I20260927 23:08:48.356412 24367 log.cc:1196] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: Deleting WAL directory at /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790550224434615-4035-0/minicluster-data/ts-2/wal/wals/6df95671d9dd4c23805596bd1ae50ca6
I20260927 23:08:48.357016 24367 ts_tablet_manager.cc:1939] T 6df95671d9dd4c23805596bd1ae50ca6 P f4ac24cb73ac46af94843a702b8651d1: Deleting consensus metadata
I20260927 23:08:48.358258 23863 catalog_manager.cc:4841] TS 84cfdfda5501416ea8c239494f0ffe56 (127.3.240.194:43971): tablet 6df95671d9dd4c23805596bd1ae50ca6 (table test-table [id=249e5db8670b4cdcb5fe304ce86f1413]) successfully deleted
I20260927 23:08:48.358750 23862 catalog_manager.cc:4841] TS f4ac24cb73ac46af94843a702b8651d1 (127.3.240.195:42441): tablet 6df95671d9dd4c23805596bd1ae50ca6 (table test-table [id=249e5db8670b4cdcb5fe304ce86f1413]) successfully deleted
I20260927 23:08:48.437439 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 23935
W20260927 23:08:48.454257 23860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:33237 (error 108)
I20260927 23:08:48.454495 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24066
I20260927 23:08:48.456995 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.456460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.0.0.1:42706)
0927 23:08:48.456684 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.456814 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260927 23:08:48.457265 23860 catalog_manager.cc:4584] TS a04558c4f9ff42caae0a40529d34927f (127.3.240.193:33237): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6df95671d9dd4c23805596bd1ae50ca6: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33237: connect: Connection refused (error 111)
I20260927 23:08:48.461758 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.455300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33086 (local address 127.3.240.194:43971)
0927 23:08:48.455809 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.455814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.455829 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:08:48.455879 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.455882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.455953 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.456036 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:48.456583 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.457459 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.458633 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.459121 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.459258 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.460376 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:48.460384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:48.460397 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:48.460975 (+ 578us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:48.461020 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:48.461039 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:48.461159 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.461311 (+ 152us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.461404 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260927 23:08:48.462523 24188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:48.510850 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.510278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.0.0.1:42718)
0927 23:08:48.510536 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.510663 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260927 23:08:48.511896 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.511516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.0.0.1:42722)
0927 23:08:48.511678 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.511755 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260927 23:08:48.569442 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 24066
I20260927 23:08:48.572557 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.572089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.0.0.1:42736)
0927 23:08:48.572314 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.572399 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260927 23:08:48.573612 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.573311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.0.0.1:42748)
0927 23:08:48.573436 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.573501 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260927 23:08:48.588198 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24196
I20260927 23:08:48.599237 24339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.588972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.3.240.195:42441)
0927 23:08:48.589135 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:48.589140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:48.589157 (+ 17us) server_negotiation.cc:407] Connection header received
0927 23:08:48.589266 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:48.589270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:48.589353 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:48.589433 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:48.589775 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.590471 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.591680 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:48.592087 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:48.592174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:48.597946 (+ 5772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:48.597955 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:48.597970 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:48.598511 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:48.598568 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:48.598587 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:48.598681 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:48.598798 (+ 117us) server_negotiation.cc:299] Negotiation successful
0927 23:08:48.598881 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:08:48.599917 24318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:48.654299 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.653666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.0.0.1:42754)
0927 23:08:48.653938 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.654109 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260927 23:08:48.655378 24337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:48.654924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33237 (local address 127.0.0.1:42760)
0927 23:08:48.655148 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 23:08:48.655239 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260927 23:08:48.688865 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 24196
I20260927 23:08:48.705932 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 23845
I20260927 23:08:48.706934 23906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:48.810250 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 23845
2026-09-27T23:08:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2616 ms)
[----------] 6 tests from EncryptionPolicyTest (20606 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:48 dist-test-slave-lmfp krb5kdc[24376](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:48 dist-test-slave-lmfp krb5kdc[24376](info): set up 2 sockets
Sep 27 23:08:48 dist-test-slave-lmfp krb5kdc[24376](info): commencing operation
krb5kdc: starting...
W20260927 23:08:49.887391 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.011s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:50 dist-test-slave-lmfp krb5kdc[24376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:08:50.122586 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:50.446473 24389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:50.446791 24389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:50.446959 24389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:50.454154 24389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:50.454293 24389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:50.454403 24389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:50.454514 24389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:50.454603 24389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:50.464303 24389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:50.465528 24389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:50.466821 24389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:50.478425 24395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:50.478377 24394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:50.478984 24397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:50.479132 24389 server_base.cc:1034] running on GCE node
I20260927 23:08:50.479959 24389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:50.481249 24389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:50.482486 24389 hybrid_clock.cc:648] HybridClock initialized: now 1790550530482440 us; error 52 us; skew 500 ppm
I20260927 23:08:50.482887 24389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:50 dist-test-slave-lmfp krb5kdc[24376](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:50.485939 24389 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:08:50.487454 24389 webserver.cc:460] Webserver started at http://127.0.0.1:40385/ using document root <none> and password file <none>
I20260927 23:08:50.488123 24389 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:50.488267 24389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:50.488582 24389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:50.491535 24389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "b330a485893c4507b5c8decceae06927"
format_stamp: "Formatted at 2026-09-27 23:08:50 on dist-test-slave-lmfp"
server_key: "7fb0ddc63c9e970b7a54d773721d397e"
server_key_iv: "996739d01e8134fda35b0d55211376f8"
server_key_version: "encryptionkey@0"
I20260927 23:08:50.492312 24389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "b330a485893c4507b5c8decceae06927"
format_stamp: "Formatted at 2026-09-27 23:08:50 on dist-test-slave-lmfp"
server_key: "7fb0ddc63c9e970b7a54d773721d397e"
server_key_iv: "996739d01e8134fda35b0d55211376f8"
server_key_version: "encryptionkey@0"
I20260927 23:08:50.497778 24389 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260927 23:08:50.501212 24404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:50.502630 24389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260927 23:08:50.502847 24389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "b330a485893c4507b5c8decceae06927"
format_stamp: "Formatted at 2026-09-27 23:08:50 on dist-test-slave-lmfp"
server_key: "7fb0ddc63c9e970b7a54d773721d397e"
server_key_iv: "996739d01e8134fda35b0d55211376f8"
server_key_version: "encryptionkey@0"
I20260927 23:08:50.503118 24389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:50.541404 24389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:50.545913 24389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:50.546241 24389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:50.566058 24389 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44345
I20260927 23:08:50.566090 24455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44345 every 8 connection(s)
I20260927 23:08:50.567559 24389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:50.571359 24456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:50.574668 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 24389
I20260927 23:08:50.574874 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:50.575687 4035 external_mini_cluster.cc:1428] Setting key 559af7ec16b4bd21507efd5958371354
I20260927 23:08:50.584460 24456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927: Bootstrap starting.
I20260927 23:08:50.588318 24456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:50.589665 24456 log.cc:824] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927: Log is configured to *not* fsync() on all Append() calls
Sep 27 23:08:50 dist-test-slave-lmfp krb5kdc[24376](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 23:08:50.592520 24456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927: No bootstrap required, opened a new log
I20260927 23:08:50.599543 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:50.577715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57445 (local address 127.0.0.1:44345)
0927 23:08:50.578622 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:50.578641 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:50.578695 (+ 54us) server_negotiation.cc:407] Connection header received
0927 23:08:50.580385 (+ 1690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:50.580407 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:50.580810 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:50.581140 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:50.583063 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:50.584384 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:50.586185 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:50.586887 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:50.587100 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:50.592021 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:50.592029 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:50.592044 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:50.592085 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:50.595396 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:50.596174 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:50.596180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:50.596186 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:50.596342 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:50.596836 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:50.596839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:50.596841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:50.597317 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:50.597682 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:50.598198 (+ 516us) server_negotiation.cc:299] Negotiation successful
0927 23:08:50.598520 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":428,"threads_started":1}
I20260927 23:08:50.600431 24456 raft_consensus.cc:348] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b330a485893c4507b5c8decceae06927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260927 23:08:50.600827 24456 raft_consensus.cc:374] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:50.600941 24456 raft_consensus.cc:729] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b330a485893c4507b5c8decceae06927, State: Initialized, Role: FOLLOWER
I20260927 23:08:50.601609 24456 consensus_queue.cc:260] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b330a485893c4507b5c8decceae06927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260927 23:08:50.601876 24456 raft_consensus.cc:388] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:50.602025 24456 raft_consensus.cc:482] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:50.602222 24456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:50.603816 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:08:50.605263 24456 raft_consensus.cc:504] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b330a485893c4507b5c8decceae06927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260927 23:08:50.605784 24456 leader_election.cc:302] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b330a485893c4507b5c8decceae06927; no voters:
I20260927 23:08:50.606416 24456 leader_election.cc:288] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:50.606570 24461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:50.607218 24461 raft_consensus.cc:686] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [term 1 LEADER]: Becoming Leader. State: Replica: b330a485893c4507b5c8decceae06927, State: Running, Role: LEADER
I20260927 23:08:50.607761 24461 consensus_queue.cc:237] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b330a485893c4507b5c8decceae06927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260927 23:08:50.609527 24461 sys_catalog.cc:455] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b330a485893c4507b5c8decceae06927. Latest consensus state: current_term: 1 leader_uuid: "b330a485893c4507b5c8decceae06927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b330a485893c4507b5c8decceae06927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } } }
I20260927 23:08:50.609787 24461 sys_catalog.cc:458] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:50.610330 24463 sys_catalog.cc:455] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b330a485893c4507b5c8decceae06927" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b330a485893c4507b5c8decceae06927" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } } }
I20260927 23:08:50.610694 24463 sys_catalog.cc:458] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:50.619055 24456 sys_catalog.cc:564] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:50.619289 24466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:50.624336 24466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:50.638856 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:50.618964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50339 (local address 127.0.0.1:44345)
0927 23:08:50.619173 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:50.619182 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:50.619774 (+ 592us) server_negotiation.cc:407] Connection header received
0927 23:08:50.619958 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:50.619966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:50.620059 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:50.620187 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:50.622561 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:50.624193 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:50.626619 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:50.627669 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:50.627800 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:50.630781 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:50.630786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:50.630789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:50.630852 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:50.634241 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:50.635521 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:50.635527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:50.635536 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:50.635725 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:50.636419 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:50.636427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:50.636433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:50.636965 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:50.637248 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:50.638319 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0927 23:08:50.638486 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260927 23:08:50.640082 24466 catalog_manager.cc:1347] Generated new cluster ID: 33078313ede2430194e8867e994c71c9
I20260927 23:08:50.640178 24466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:50.658640 24466 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:50.659947 24466 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:50.666936 24466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b330a485893c4507b5c8decceae06927: Generated new TSK 0
I20260927 23:08:50.667748 24466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:50.683079 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24389
I20260927 23:08:50.690411 24451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:50.780750 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 24389
W20260927 23:08:50.797534 24374 connection.cc:535] client connection to 127.0.0.1:44345 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44345): Connection reset by peer (error 104)
2026-09-27T23:08:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:50 dist-test-slave-lmfp krb5kdc[24488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:50 dist-test-slave-lmfp krb5kdc[24488](info): set up 2 sockets
Sep 27 23:08:50 dist-test-slave-lmfp krb5kdc[24488](info): commencing operation
krb5kdc: starting...
W20260927 23:08:51.847221 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:51 dist-test-slave-lmfp krb5kdc[24488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:08:52.074205 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36945
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:52.390828 24501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:52.391170 24501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:52.391340 24501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:52.398227 24501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:52.398319 24501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:52.398427 24501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:52.398530 24501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:52.398610 24501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:52.407896 24501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36945
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:52.409132 24501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:52.410406 24501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:52.421509 24506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:52.421468 24509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:52.421557 24507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:52.422230 24501 server_base.cc:1034] running on GCE node
I20260927 23:08:52.422912 24501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:52.424206 24501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:52.425436 24501 hybrid_clock.cc:648] HybridClock initialized: now 1790550532425400 us; error 43 us; skew 500 ppm
I20260927 23:08:52.425750 24501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:52 dist-test-slave-lmfp krb5kdc[24488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:52.428568 24501 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:08:52.429876 24501 webserver.cc:460] Webserver started at http://127.0.0.1:35577/ using document root <none> and password file <none>
I20260927 23:08:52.430538 24501 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:52.430677 24501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:52.430964 24501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:52.433689 24501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "9ef2664f281c4e4181e5607cf76ac26d"
format_stamp: "Formatted at 2026-09-27 23:08:52 on dist-test-slave-lmfp"
server_key: "fc4d4461a06c60fb69d4576748e271fb"
server_key_iv: "677594811652914a53f130161352e6a8"
server_key_version: "encryptionkey@0"
I20260927 23:08:52.434454 24501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "9ef2664f281c4e4181e5607cf76ac26d"
format_stamp: "Formatted at 2026-09-27 23:08:52 on dist-test-slave-lmfp"
server_key: "fc4d4461a06c60fb69d4576748e271fb"
server_key_iv: "677594811652914a53f130161352e6a8"
server_key_version: "encryptionkey@0"
I20260927 23:08:52.439643 24501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260927 23:08:52.442894 24516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:52.444176 24501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:08:52.444356 24501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "9ef2664f281c4e4181e5607cf76ac26d"
format_stamp: "Formatted at 2026-09-27 23:08:52 on dist-test-slave-lmfp"
server_key: "fc4d4461a06c60fb69d4576748e271fb"
server_key_iv: "677594811652914a53f130161352e6a8"
server_key_version: "encryptionkey@0"
I20260927 23:08:52.444546 24501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:52.473881 24501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:52.477908 24501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:52.478196 24501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:52.496914 24501 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36945
I20260927 23:08:52.496948 24567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36945 every 8 connection(s)
I20260927 23:08:52.498193 24501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:52.501830 24568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:52.506757 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 24501
I20260927 23:08:52.506934 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:52.507464 4035 external_mini_cluster.cc:1428] Setting key d6676e4b8a464ad143fe7d4d62c85bd1
I20260927 23:08:52.513576 24568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d: Bootstrap starting.
I20260927 23:08:52.517231 24568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:52.518330 24568 log.cc:824] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:52.521657 24568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d: No bootstrap required, opened a new log
I20260927 23:08:52.523433 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:52.509467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54143 (local address 127.0.0.1:36945)
0927 23:08:52.510545 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:52.510560 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:52.510876 (+ 316us) server_negotiation.cc:407] Connection header received
0927 23:08:52.513280 (+ 2404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:52.513309 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:52.513807 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:52.514251 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:52.514733 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:52.516640 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:52.517986 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:52.519051 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:52.519197 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:52.520538 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:52.520548 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:52.520571 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:52.521257 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:52.521359 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:52.521390 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:52.521542 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:52.521949 (+ 407us) server_negotiation.cc:299] Negotiation successful
0927 23:08:52.522140 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":488,"threads_started":1}
I20260927 23:08:52.527287 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:08:52.531231 24568 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef2664f281c4e4181e5607cf76ac26d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260927 23:08:52.531677 24568 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:52.531783 24568 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef2664f281c4e4181e5607cf76ac26d, State: Initialized, Role: FOLLOWER
I20260927 23:08:52.532454 24568 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef2664f281c4e4181e5607cf76ac26d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260927 23:08:52.532761 24568 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:52.532893 24568 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:52.533083 24568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:52.535908 24568 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef2664f281c4e4181e5607cf76ac26d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260927 23:08:52.536392 24568 leader_election.cc:302] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ef2664f281c4e4181e5607cf76ac26d; no voters:
I20260927 23:08:52.537031 24568 leader_election.cc:288] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:52.537150 24578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:52.538424 24578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [term 1 LEADER]: Becoming Leader. State: Replica: 9ef2664f281c4e4181e5607cf76ac26d, State: Running, Role: LEADER
I20260927 23:08:52.539296 24571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:52.532409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49297 (local address 127.0.0.1:36945)
0927 23:08:52.532574 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:52.532580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:52.532968 (+ 388us) server_negotiation.cc:407] Connection header received
0927 23:08:52.533160 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:52.533165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:52.533243 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:52.533342 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:52.533791 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:52.534899 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:52.536172 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:52.536673 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:52.536804 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:52.538117 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:52.538126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:52.538134 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:52.538161 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:52.538392 (+ 231us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:52.538425 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:52.538664 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:52.538901 (+ 237us) server_negotiation.cc:299] Negotiation successful
0927 23:08:52.538959 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:08:52.539238 24578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef2664f281c4e4181e5607cf76ac26d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } }
I20260927 23:08:52.539960 24568 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:52.542073 24580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ef2664f281c4e4181e5607cf76ac26d. Latest consensus state: current_term: 1 leader_uuid: "9ef2664f281c4e4181e5607cf76ac26d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef2664f281c4e4181e5607cf76ac26d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } } }
I20260927 23:08:52.542445 24580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:52.543206 24579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ef2664f281c4e4181e5607cf76ac26d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ef2664f281c4e4181e5607cf76ac26d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36945 } } }
I20260927 23:08:52.543555 24579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:52.544955 24585 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:52.549964 24585 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:52.559197 24585 catalog_manager.cc:1347] Generated new cluster ID: a09ffecbb78c41c2bc1f44f7f834c8c2
I20260927 23:08:52.559298 24585 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:52.573957 24585 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:52.575722 24585 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:52.594321 24585 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ef2664f281c4e4181e5607cf76ac26d: Generated new TSK 0
I20260927 23:08:52.595400 24585 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:52.610224 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24501
W20260927 23:08:52.611397 24563 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54143
W20260927 23:08:52.612093 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24501: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:08:52.612289 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 24501
W20260927 23:08:52.628907 24486 connection.cc:535] client connection to 127.0.0.1:36945 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36945): Connection reset by peer (error 104)
2026-09-27T23:08:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:52 dist-test-slave-lmfp krb5kdc[24599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:52 dist-test-slave-lmfp krb5kdc[24599](info): set up 2 sockets
Sep 27 23:08:52 dist-test-slave-lmfp krb5kdc[24599](info): commencing operation
krb5kdc: starting...
W20260927 23:08:53.687114 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:53 dist-test-slave-lmfp krb5kdc[24599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:08:53.925777 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:54.249785 24612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:54.250108 24612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:54.250278 24612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:54.257256 24612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:54.257404 24612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:54.257506 24612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:54.257627 24612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:54.257706 24612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:54.267621 24612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:54.268841 24612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:54.270118 24612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:54.281697 24618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:54.281571 24620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:54.281728 24617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:54.282373 24612 server_base.cc:1034] running on GCE node
I20260927 23:08:54.283190 24612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:54.284387 24612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:54.285571 24612 hybrid_clock.cc:648] HybridClock initialized: now 1790550534285536 us; error 24 us; skew 500 ppm
I20260927 23:08:54.285918 24612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:54 dist-test-slave-lmfp krb5kdc[24599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:54.294566 24612 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:08:54.296283 24612 webserver.cc:460] Webserver started at http://127.0.0.1:37411/ using document root <none> and password file <none>
I20260927 23:08:54.297008 24612 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:54.297134 24612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:54.297477 24612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:54.300535 24612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "96476cab837b48c0aa5224907dd31bf5"
format_stamp: "Formatted at 2026-09-27 23:08:54 on dist-test-slave-lmfp"
server_key: "14e3cbafc1c1500a6bc0bf23e45d4082"
server_key_iv: "a2f53d48997d851033505b56c2f749a5"
server_key_version: "encryptionkey@0"
I20260927 23:08:54.301247 24612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "96476cab837b48c0aa5224907dd31bf5"
format_stamp: "Formatted at 2026-09-27 23:08:54 on dist-test-slave-lmfp"
server_key: "14e3cbafc1c1500a6bc0bf23e45d4082"
server_key_iv: "a2f53d48997d851033505b56c2f749a5"
server_key_version: "encryptionkey@0"
I20260927 23:08:54.306452 24612 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 23:08:54.310201 24627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:54.311633 24612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:08:54.311833 24612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "96476cab837b48c0aa5224907dd31bf5"
format_stamp: "Formatted at 2026-09-27 23:08:54 on dist-test-slave-lmfp"
server_key: "14e3cbafc1c1500a6bc0bf23e45d4082"
server_key_iv: "a2f53d48997d851033505b56c2f749a5"
server_key_version: "encryptionkey@0"
I20260927 23:08:54.312048 24612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:54.317988 24612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:54.322175 24612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:54.322484 24612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:54.342348 24612 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40101
I20260927 23:08:54.342386 24678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40101 every 8 connection(s)
I20260927 23:08:54.344041 24612 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:54.347707 24679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:54.348580 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 24612
I20260927 23:08:54.348768 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:54.349323 4035 external_mini_cluster.cc:1428] Setting key 3ec9e185ebeb7a2041ea9509ce776aa8
I20260927 23:08:54.357039 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:54.351149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45169 (local address 127.0.0.1:40101)
0927 23:08:54.351954 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:54.351965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:54.352004 (+ 39us) server_negotiation.cc:407] Connection header received
0927 23:08:54.353798 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:54.353820 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:54.354159 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:54.354482 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:54.355519 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:54.355524 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:54.355537 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:54.355549 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:54.355602 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:54.355622 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:54.355796 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:54.355942 (+ 146us) server_negotiation.cc:299] Negotiation successful
0927 23:08:54.356145 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":379,"threads_started":1}
I20260927 23:08:54.360450 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:08:54.360469 24679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5: Bootstrap starting.
I20260927 23:08:54.364071 24679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:54.365370 24679 log.cc:824] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:54.368072 24682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:54.365559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40925 (local address 127.0.0.1:40101)
0927 23:08:54.365723 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:54.365729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:54.366125 (+ 396us) server_negotiation.cc:407] Connection header received
0927 23:08:54.366306 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:54.366311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:54.366369 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:54.366471 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:54.367428 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:54.367433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:54.367437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:54.367452 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:54.367494 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:54.367507 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:54.367629 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:54.367764 (+ 135us) server_negotiation.cc:299] Negotiation successful
0927 23:08:54.367818 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:08:54.368418 24679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5: No bootstrap required, opened a new log
I20260927 23:08:54.375771 24679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96476cab837b48c0aa5224907dd31bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260927 23:08:54.376118 24679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:54.376219 24679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96476cab837b48c0aa5224907dd31bf5, State: Initialized, Role: FOLLOWER
I20260927 23:08:54.376713 24679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96476cab837b48c0aa5224907dd31bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260927 23:08:54.376957 24679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:54.377099 24679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:54.377262 24679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:54.379670 24679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96476cab837b48c0aa5224907dd31bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260927 23:08:54.380152 24679 leader_election.cc:302] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96476cab837b48c0aa5224907dd31bf5; no voters:
I20260927 23:08:54.380707 24679 leader_election.cc:288] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:54.380851 24689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:54.381493 24689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [term 1 LEADER]: Becoming Leader. State: Replica: 96476cab837b48c0aa5224907dd31bf5, State: Running, Role: LEADER
I20260927 23:08:54.381953 24689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96476cab837b48c0aa5224907dd31bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260927 23:08:54.382618 24679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:54.384553 24690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96476cab837b48c0aa5224907dd31bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96476cab837b48c0aa5224907dd31bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } } }
I20260927 23:08:54.384997 24690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:54.384892 24691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96476cab837b48c0aa5224907dd31bf5. Latest consensus state: current_term: 1 leader_uuid: "96476cab837b48c0aa5224907dd31bf5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96476cab837b48c0aa5224907dd31bf5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } } }
I20260927 23:08:54.385274 24691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:54.388077 24695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:54.392166 24695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:54.400437 24695 catalog_manager.cc:1347] Generated new cluster ID: cf137bd7f34f49658c32e475b1b14a2d
I20260927 23:08:54.400537 24695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:54.410425 24695 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:54.410771 24695 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:54.415760 24695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96476cab837b48c0aa5224907dd31bf5: Generated new TSK 0
I20260927 23:08:54.416656 24695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:54.428912 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24612
W20260927 23:08:54.429806 24674 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45169
W20260927 23:08:54.430460 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24612: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:08:54.430600 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 24612
2026-09-27T23:08:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1817 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:54 dist-test-slave-lmfp krb5kdc[24710](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:54 dist-test-slave-lmfp krb5kdc[24710](info): set up 2 sockets
Sep 27 23:08:54 dist-test-slave-lmfp krb5kdc[24710](info): commencing operation
krb5kdc: starting...
W20260927 23:08:55.495409 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:55 dist-test-slave-lmfp krb5kdc[24710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:08:55.722440 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:56.034512 24723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:56.034772 24723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:56.034955 24723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:56.041889 24723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:56.041989 24723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:56.042098 24723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:56.042191 24723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:56.042272 24723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:56.051608 24723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:56.052819 24723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:56.054060 24723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:56.065999 24728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:56.065972 24731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:56.066010 24729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:56.066819 24723 server_base.cc:1034] running on GCE node
I20260927 23:08:56.067617 24723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:56.068990 24723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:56.070215 24723 hybrid_clock.cc:648] HybridClock initialized: now 1790550536070190 us; error 59 us; skew 500 ppm
I20260927 23:08:56.070545 24723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:56 dist-test-slave-lmfp krb5kdc[24710](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:56.073495 24723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:08:56.074898 24723 webserver.cc:460] Webserver started at http://127.0.0.1:38669/ using document root <none> and password file <none>
I20260927 23:08:56.075591 24723 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:56.075696 24723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:56.075950 24723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:56.078631 24723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "75a32f77a521487a8c7df297ef2561e9"
format_stamp: "Formatted at 2026-09-27 23:08:56 on dist-test-slave-lmfp"
server_key: "d7b690d0c490505707839edd68aeeb04"
server_key_iv: "c89d6bab19c6730614ed576f08fbe670"
server_key_version: "encryptionkey@0"
I20260927 23:08:56.079345 24723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "75a32f77a521487a8c7df297ef2561e9"
format_stamp: "Formatted at 2026-09-27 23:08:56 on dist-test-slave-lmfp"
server_key: "d7b690d0c490505707839edd68aeeb04"
server_key_iv: "c89d6bab19c6730614ed576f08fbe670"
server_key_version: "encryptionkey@0"
I20260927 23:08:56.084093 24723 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260927 23:08:56.087116 24738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:56.088321 24723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260927 23:08:56.088519 24723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "75a32f77a521487a8c7df297ef2561e9"
format_stamp: "Formatted at 2026-09-27 23:08:56 on dist-test-slave-lmfp"
server_key: "d7b690d0c490505707839edd68aeeb04"
server_key_iv: "c89d6bab19c6730614ed576f08fbe670"
server_key_version: "encryptionkey@0"
I20260927 23:08:56.088718 24723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:56.109012 24723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:56.112516 24723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:56.112813 24723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:56.130235 24723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45383
I20260927 23:08:56.130311 24789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45383 every 8 connection(s)
I20260927 23:08:56.131870 24723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:56.133894 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 24723
I20260927 23:08:56.134090 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:56.134756 4035 external_mini_cluster.cc:1428] Setting key fd9cbafaeeba7a7d2da9b4f74284c12e
I20260927 23:08:56.138935 24790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 27 23:08:56 dist-test-slave-lmfp krb5kdc[24710](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 23:08:56.151816 24790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9: Bootstrap starting.
I20260927 23:08:56.154839 24790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:56.155887 24790 log.cc:824] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:56.159096 24790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9: No bootstrap required, opened a new log
I20260927 23:08:56.159482 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:56.137255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59081 (local address 127.0.0.1:45383)
0927 23:08:56.138100 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:56.138115 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:56.138176 (+ 61us) server_negotiation.cc:407] Connection header received
0927 23:08:56.140750 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:56.140778 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:56.141252 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:56.141581 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:56.143407 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:56.144726 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:56.146478 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:56.147274 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:56.147421 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:56.152117 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:56.152125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:56.152139 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:56.152174 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:56.155147 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:56.155822 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:56.155829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:56.155837 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:56.156036 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:56.156514 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:56.156524 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:56.156529 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:56.157098 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:56.157379 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:56.157750 (+ 371us) server_negotiation.cc:299] Negotiation successful
0927 23:08:56.158170 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":358,"threads_started":1}
I20260927 23:08:56.163275 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:08:56.169137 24790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a32f77a521487a8c7df297ef2561e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260927 23:08:56.169514 24790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:56.169644 24790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a32f77a521487a8c7df297ef2561e9, State: Initialized, Role: FOLLOWER
I20260927 23:08:56.170528 24790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a32f77a521487a8c7df297ef2561e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260927 23:08:56.170853 24800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260927 23:08:56.171121 24800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a32f77a521487a8c7df297ef2561e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260927 23:08:56.170827 24790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:56.171759 24790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:56.171706 24800 leader_election.cc:302] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a32f77a521487a8c7df297ef2561e9; no voters:
I20260927 23:08:56.171875 24790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:56.172492 24800 leader_election.cc:288] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260927 23:08:56.174280 24790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a32f77a521487a8c7df297ef2561e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260927 23:08:56.174518 24790 leader_election.cc:302] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a32f77a521487a8c7df297ef2561e9; no voters:
I20260927 23:08:56.174559 24801 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260927 23:08:56.174724 24790 leader_election.cc:288] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:56.176052 24801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a32f77a521487a8c7df297ef2561e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } } }
I20260927 23:08:56.176496 24801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:08:56.176707 24800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:56.176872 24800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [term 1 LEADER]: Becoming Leader. State: Replica: 75a32f77a521487a8c7df297ef2561e9, State: Running, Role: LEADER
I20260927 23:08:56.177474 24800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a32f77a521487a8c7df297ef2561e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } }
I20260927 23:08:56.181983 24793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:56.168331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39987 (local address 127.0.0.1:45383)
0927 23:08:56.168513 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:56.168518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:56.168881 (+ 363us) server_negotiation.cc:407] Connection header received
0927 23:08:56.168988 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:56.168993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:56.169056 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:56.169189 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:56.170618 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:56.171567 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:56.173094 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:56.173544 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:56.173688 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:56.176064 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:56.176069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:56.176073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:08:56.176107 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:56.179208 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:56.179925 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:56.179932 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:56.179936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:56.180088 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:08:56.180462 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:08:56.180468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:08:56.180472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:08:56.180748 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:56.181022 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:56.181348 (+ 326us) server_negotiation.cc:299] Negotiation successful
0927 23:08:56.181568 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260927 23:08:56.182971 24801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a32f77a521487a8c7df297ef2561e9. Latest consensus state: current_term: 1 leader_uuid: "75a32f77a521487a8c7df297ef2561e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a32f77a521487a8c7df297ef2561e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45383 } } }
I20260927 23:08:56.183238 24801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:56.185537 24790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:56.185760 24804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:56.190276 24804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:56.201864 24804 catalog_manager.cc:1347] Generated new cluster ID: 7f8eaf2c448f45dc86f7f60d8bfafffc
I20260927 23:08:56.201974 24804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:56.221014 24804 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:56.222268 24804 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:56.229861 24804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75a32f77a521487a8c7df297ef2561e9: Generated new TSK 0
I20260927 23:08:56.230589 24804 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:56.243177 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24723
I20260927 23:08:56.244203 24785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:08:56.333016 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 24723
2026-09-27T23:08:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1902 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:56 dist-test-slave-lmfp krb5kdc[24822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:56 dist-test-slave-lmfp krb5kdc[24822](info): set up 2 sockets
Sep 27 23:08:56 dist-test-slave-lmfp krb5kdc[24822](info): commencing operation
krb5kdc: starting...
W20260927 23:08:57.395362 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:57 dist-test-slave-lmfp krb5kdc[24822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:08:57.621665 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:57.934568 24835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:57.934916 24835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:57.935108 24835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:57.942173 24835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:57.942314 24835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:57.942438 24835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:57.942538 24835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:57.942617 24835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:57.952625 24835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:57.953984 24835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:57.955358 24835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:57.966813 24840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:57.966771 24843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:57.966873 24841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:57.967638 24835 server_base.cc:1034] running on GCE node
I20260927 23:08:57.968429 24835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:57.969849 24835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:57.971071 24835 hybrid_clock.cc:648] HybridClock initialized: now 1790550537971010 us; error 47 us; skew 500 ppm
I20260927 23:08:57.971416 24835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:57 dist-test-slave-lmfp krb5kdc[24822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:57.974414 24835 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:08:57.975741 24835 webserver.cc:460] Webserver started at http://127.0.0.1:43935/ using document root <none> and password file <none>
I20260927 23:08:57.976346 24835 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:57.976439 24835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:57.976670 24835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:57.979552 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "a387a190bd29401991218647d763dc4d"
format_stamp: "Formatted at 2026-09-27 23:08:57 on dist-test-slave-lmfp"
server_key: "5205f6f39df68aa4c76882a36a00beae"
server_key_iv: "f632aa07287b918d4de4aff5c451bc56"
server_key_version: "encryptionkey@0"
I20260927 23:08:57.980253 24835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "a387a190bd29401991218647d763dc4d"
format_stamp: "Formatted at 2026-09-27 23:08:57 on dist-test-slave-lmfp"
server_key: "5205f6f39df68aa4c76882a36a00beae"
server_key_iv: "f632aa07287b918d4de4aff5c451bc56"
server_key_version: "encryptionkey@0"
I20260927 23:08:57.985170 24835 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:08:57.988374 24850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:57.989616 24835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260927 23:08:57.989789 24835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "a387a190bd29401991218647d763dc4d"
format_stamp: "Formatted at 2026-09-27 23:08:57 on dist-test-slave-lmfp"
server_key: "5205f6f39df68aa4c76882a36a00beae"
server_key_iv: "f632aa07287b918d4de4aff5c451bc56"
server_key_version: "encryptionkey@0"
I20260927 23:08:57.989969 24835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:58.009570 24835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:58.013275 24835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:58.013530 24835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:58.031735 24835 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34547
I20260927 23:08:58.031778 24901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34547 every 8 connection(s)
I20260927 23:08:58.033172 24835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:58.033685 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 24835
I20260927 23:08:58.033900 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:58.034473 4035 external_mini_cluster.cc:1428] Setting key 782fdcd9b7dca08eed42a889402a9484
I20260927 23:08:58.037640 24902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:58.047626 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:58.036271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37647 (local address 127.0.0.1:34547)
0927 23:08:58.037081 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:58.037093 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:58.037140 (+ 47us) server_negotiation.cc:407] Connection header received
0927 23:08:58.038833 (+ 1693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:58.038853 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:58.039285 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:58.039695 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:58.040249 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:58.041591 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:58.043045 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:58.043723 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:58.043926 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:58.045160 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:58.045168 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:58.045181 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:58.045815 (+ 634us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:58.045898 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:58.045929 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:58.046119 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:58.046352 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 23:08:58.046604 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":380,"threads_started":1}
I20260927 23:08:58.048053 24902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d: Bootstrap starting.
I20260927 23:08:58.051335 24902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:58.051373 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:08:58.052268 24902 log.cc:824] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:58.054911 24902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d: No bootstrap required, opened a new log
I20260927 23:08:58.062331 24902 raft_consensus.cc:348] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387a190bd29401991218647d763dc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260927 23:08:58.062742 24902 raft_consensus.cc:374] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:58.062855 24902 raft_consensus.cc:729] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a387a190bd29401991218647d763dc4d, State: Initialized, Role: FOLLOWER
I20260927 23:08:58.063529 24902 consensus_queue.cc:260] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387a190bd29401991218647d763dc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260927 23:08:58.063824 24902 raft_consensus.cc:388] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:58.063987 24902 raft_consensus.cc:482] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:58.064157 24902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:58.065822 24905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:58.055988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54777 (local address 127.0.0.1:34547)
0927 23:08:58.056169 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:58.056176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:58.059573 (+ 3397us) server_negotiation.cc:407] Connection header received
0927 23:08:58.059739 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:58.059744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:58.059823 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:58.059923 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:58.060448 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:58.061525 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:58.062974 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:08:58.063658 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:08:58.063776 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:08:58.065094 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:58.065101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:58.065107 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:58.065136 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:58.065183 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:58.065202 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:58.065302 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:58.065398 (+ 96us) server_negotiation.cc:299] Negotiation successful
0927 23:08:58.065482 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:08:58.066882 24902 raft_consensus.cc:504] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387a190bd29401991218647d763dc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260927 23:08:58.067381 24902 leader_election.cc:302] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a387a190bd29401991218647d763dc4d; no voters:
I20260927 23:08:58.067970 24902 leader_election.cc:288] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:58.068065 24912 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:58.068849 24912 raft_consensus.cc:686] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [term 1 LEADER]: Becoming Leader. State: Replica: a387a190bd29401991218647d763dc4d, State: Running, Role: LEADER
I20260927 23:08:58.069337 24912 consensus_queue.cc:237] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387a190bd29401991218647d763dc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } }
I20260927 23:08:58.069980 24902 sys_catalog.cc:564] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:58.072038 24913 sys_catalog.cc:455] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a387a190bd29401991218647d763dc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387a190bd29401991218647d763dc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } } }
I20260927 23:08:58.072415 24913 sys_catalog.cc:458] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:58.072321 24914 sys_catalog.cc:455] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a387a190bd29401991218647d763dc4d. Latest consensus state: current_term: 1 leader_uuid: "a387a190bd29401991218647d763dc4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a387a190bd29401991218647d763dc4d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34547 } } }
I20260927 23:08:58.072595 24914 sys_catalog.cc:458] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:58.073385 24920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:58.079478 24920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:58.089032 24920 catalog_manager.cc:1347] Generated new cluster ID: f448e28e42ad4ccdad4909ff987985cc
I20260927 23:08:58.089144 24920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:58.122623 24920 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:58.123919 24920 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:58.131068 24920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a387a190bd29401991218647d763dc4d: Generated new TSK 0
I20260927 23:08:58.131857 24920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:58.147505 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24835
W20260927 23:08:58.148604 24897 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37647
W20260927 23:08:58.149304 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24835: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:08:58.149437 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 24835
2026-09-27T23:08:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1816 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:08:58 dist-test-slave-lmfp krb5kdc[24933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:08:58 dist-test-slave-lmfp krb5kdc[24933](info): set up 2 sockets
Sep 27 23:08:58 dist-test-slave-lmfp krb5kdc[24933](info): commencing operation
krb5kdc: starting...
W20260927 23:08:59.195240 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:08:59 dist-test-slave-lmfp krb5kdc[24933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:08:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:08:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:08:59.422794 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:08:59.737931 24946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:08:59.738212 24946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:08:59.738368 24946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:08:59.745363 24946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:08:59.745481 24946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:08:59.745575 24946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:08:59.745673 24946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:08:59.745747 24946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:08:59.755072 24946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:08:59.756251 24946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:08:59.757531 24946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:08:59.769171 24952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:59.769135 24954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:08:59.769204 24951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:08:59.770160 24946 server_base.cc:1034] running on GCE node
I20260927 23:08:59.770918 24946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:08:59.772229 24946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:08:59.773478 24946 hybrid_clock.cc:648] HybridClock initialized: now 1790550539773386 us; error 76 us; skew 500 ppm
I20260927 23:08:59.773842 24946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:08:59 dist-test-slave-lmfp krb5kdc[24933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550539, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:08:59.784121 24946 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:08:59.785552 24946 webserver.cc:460] Webserver started at http://127.0.0.1:37115/ using document root <none> and password file <none>
I20260927 23:08:59.786279 24946 fs_manager.cc:359] Metadata directory not provided
I20260927 23:08:59.786428 24946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:08:59.786752 24946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:08:59.789899 24946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "7fd4179fae86431d99f0271ef1f0aa6b"
format_stamp: "Formatted at 2026-09-27 23:08:59 on dist-test-slave-lmfp"
server_key: "2d109049a4e3b69834160a3e90fc10e7"
server_key_iv: "e3e4990653c691fe71867813fca6b067"
server_key_version: "encryptionkey@0"
I20260927 23:08:59.790714 24946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "7fd4179fae86431d99f0271ef1f0aa6b"
format_stamp: "Formatted at 2026-09-27 23:08:59 on dist-test-slave-lmfp"
server_key: "2d109049a4e3b69834160a3e90fc10e7"
server_key_iv: "e3e4990653c691fe71867813fca6b067"
server_key_version: "encryptionkey@0"
I20260927 23:08:59.795843 24946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260927 23:08:59.799412 24961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:08:59.800617 24946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260927 23:08:59.800827 24946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "7fd4179fae86431d99f0271ef1f0aa6b"
format_stamp: "Formatted at 2026-09-27 23:08:59 on dist-test-slave-lmfp"
server_key: "2d109049a4e3b69834160a3e90fc10e7"
server_key_iv: "e3e4990653c691fe71867813fca6b067"
server_key_version: "encryptionkey@0"
I20260927 23:08:59.801024 24946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:08:59.806510 24946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:08:59.810750 24946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:08:59.811085 24946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:08:59.829277 24946 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37557
I20260927 23:08:59.829316 25012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37557 every 8 connection(s)
I20260927 23:08:59.830590 24946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:08:59.834160 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 24946
I20260927 23:08:59.834369 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:08:59.834652 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:08:59.835024 4035 external_mini_cluster.cc:1428] Setting key 073aba638ec99cb21e3c2014bad63acd
I20260927 23:08:59.843146 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:59.837150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40673 (local address 127.0.0.1:37557)
0927 23:08:59.837979 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:59.837990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:59.838046 (+ 56us) server_negotiation.cc:407] Connection header received
0927 23:08:59.839812 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:59.839832 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:59.840136 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:59.840447 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:59.841579 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:59.841584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:59.841596 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:59.841614 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:59.841675 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:59.841697 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:59.841858 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:59.842017 (+ 159us) server_negotiation.cc:299] Negotiation successful
0927 23:08:59.842224 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"spinlock_wait_cycles":19328,"thread_start_us":469,"threads_started":1}
I20260927 23:08:59.846482 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:08:59.848512 25013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b: Bootstrap starting.
I20260927 23:08:59.852128 25013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b: Neither blocks nor log segments found. Creating new log.
I20260927 23:08:59.853260 25013 log.cc:824] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b: Log is configured to *not* fsync() on all Append() calls
I20260927 23:08:59.853345 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:08:59.850817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60419 (local address 127.0.0.1:37557)
0927 23:08:59.851032 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0927 23:08:59.851040 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:08:59.851410 (+ 370us) server_negotiation.cc:407] Connection header received
0927 23:08:59.851637 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:08:59.851642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:08:59.851694 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:08:59.851804 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:08:59.852719 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:08:59.852724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:08:59.852729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:08:59.852746 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:08:59.852812 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:08:59.852826 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:08:59.852938 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0927 23:08:59.853031 (+ 93us) server_negotiation.cc:299] Negotiation successful
0927 23:08:59.853089 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 23:08:59.856261 25013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b: No bootstrap required, opened a new log
I20260927 23:08:59.863794 25013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd4179fae86431d99f0271ef1f0aa6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260927 23:08:59.864185 25013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:08:59.864305 25013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fd4179fae86431d99f0271ef1f0aa6b, State: Initialized, Role: FOLLOWER
I20260927 23:08:59.864923 25013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd4179fae86431d99f0271ef1f0aa6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260927 23:08:59.865200 25013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:08:59.865356 25013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:08:59.865535 25013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:08:59.867978 25013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd4179fae86431d99f0271ef1f0aa6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260927 23:08:59.868480 25013 leader_election.cc:302] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fd4179fae86431d99f0271ef1f0aa6b; no voters:
I20260927 23:08:59.869082 25013 leader_election.cc:288] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:08:59.869237 25023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:08:59.870306 25023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [term 1 LEADER]: Becoming Leader. State: Replica: 7fd4179fae86431d99f0271ef1f0aa6b, State: Running, Role: LEADER
I20260927 23:08:59.870846 25023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd4179fae86431d99f0271ef1f0aa6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } }
I20260927 23:08:59.871313 25013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:08:59.873852 25025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fd4179fae86431d99f0271ef1f0aa6b. Latest consensus state: current_term: 1 leader_uuid: "7fd4179fae86431d99f0271ef1f0aa6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd4179fae86431d99f0271ef1f0aa6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } } }
I20260927 23:08:59.873818 25024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fd4179fae86431d99f0271ef1f0aa6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fd4179fae86431d99f0271ef1f0aa6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37557 } } }
I20260927 23:08:59.874281 25024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:59.874291 25025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b [sys.catalog]: This master's current role is: LEADER
I20260927 23:08:59.876452 25030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:08:59.881738 25030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:08:59.890002 25030 catalog_manager.cc:1347] Generated new cluster ID: 79ce6605c12e40afa1a8666f71129f8a
I20260927 23:08:59.890113 25030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:08:59.906827 25030 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:08:59.907316 25030 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:08:59.918426 25030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fd4179fae86431d99f0271ef1f0aa6b: Generated new TSK 0
I20260927 23:08:59.919474 25030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:08:59.925005 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24946
W20260927 23:08:59.926360 25008 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40673
W20260927 23:08:59.926968 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 24946: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:08:59.927139 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 24946
2026-09-27T23:08:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1777 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:00 dist-test-slave-lmfp krb5kdc[25044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:00 dist-test-slave-lmfp krb5kdc[25044](info): set up 2 sockets
Sep 27 23:09:00 dist-test-slave-lmfp krb5kdc[25044](info): commencing operation
krb5kdc: starting...
W20260927 23:09:00.987432 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:01 dist-test-slave-lmfp krb5kdc[25044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:09:01.222899 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:01.562029 25057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:01.562294 25057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:01.562498 25057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:01.569628 25057 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:01.569723 25057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:01.569801 25057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:01.569929 25057 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:01.569989 25057 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:01.579509 25057 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:01.580729 25057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:01.581990 25057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:01.593441 25065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:01.593487 25063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:01.593515 25062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:01.594477 25057 server_base.cc:1034] running on GCE node
I20260927 23:09:01.595362 25057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:01.596635 25057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:01.597822 25057 hybrid_clock.cc:648] HybridClock initialized: now 1790550541597773 us; error 45 us; skew 500 ppm
I20260927 23:09:01.598119 25057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:01 dist-test-slave-lmfp krb5kdc[25044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:01.601042 25057 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:09:01.602489 25057 webserver.cc:460] Webserver started at http://127.0.0.1:34855/ using document root <none> and password file <none>
I20260927 23:09:01.603323 25057 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:01.603466 25057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:01.603842 25057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:01.606804 25057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "70095832a550474199bca5e48e3c0309"
format_stamp: "Formatted at 2026-09-27 23:09:01 on dist-test-slave-lmfp"
server_key: "79c59271207ead4cbbf58028ea504402"
server_key_iv: "e615055d521590cdd223feae14cc844a"
server_key_version: "encryptionkey@0"
I20260927 23:09:01.607628 25057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "70095832a550474199bca5e48e3c0309"
format_stamp: "Formatted at 2026-09-27 23:09:01 on dist-test-slave-lmfp"
server_key: "79c59271207ead4cbbf58028ea504402"
server_key_iv: "e615055d521590cdd223feae14cc844a"
server_key_version: "encryptionkey@0"
I20260927 23:09:01.612552 25057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 23:09:01.615702 25072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:01.616863 25057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260927 23:09:01.617027 25057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "70095832a550474199bca5e48e3c0309"
format_stamp: "Formatted at 2026-09-27 23:09:01 on dist-test-slave-lmfp"
server_key: "79c59271207ead4cbbf58028ea504402"
server_key_iv: "e615055d521590cdd223feae14cc844a"
server_key_version: "encryptionkey@0"
I20260927 23:09:01.617194 25057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:01.647863 25057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:01.651590 25057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:01.651877 25057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:01.669793 25057 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36921
I20260927 23:09:01.669872 25123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36921 every 8 connection(s)
I20260927 23:09:01.671083 25057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:01.674625 25124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:01.676235 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25057
I20260927 23:09:01.676411 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:01.676956 4035 external_mini_cluster.cc:1428] Setting key 53efb85b0a54876691dfaa02c07a6e28
I20260927 23:09:01.688645 25124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309: Bootstrap starting.
Sep 27 23:09:01 dist-test-slave-lmfp krb5kdc[25044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 23:09:01.692193 25124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:01.693534 25124 log.cc:824] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:01.698161 25124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309: No bootstrap required, opened a new log
I20260927 23:09:01.700868 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:01.678717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:38653 (local address 127.0.0.1:36921)
0927 23:09:01.679485 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:01.679495 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:01.679538 (+ 43us) server_negotiation.cc:407] Connection header received
0927 23:09:01.680993 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:01.681016 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:01.681406 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:01.681735 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:01.683442 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:01.684661 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:01.686418 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:01.687108 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:01.687228 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:01.691940 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:01.691948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:01.691961 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:01.691995 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:01.696036 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:01.697022 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:01.697029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:01.697037 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:01.697245 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:01.697732 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:01.697737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:01.697740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:01.698418 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:01.698747 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:01.699193 (+ 446us) server_negotiation.cc:299] Negotiation successful
0927 23:09:01.699516 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":561,"thread_start_us":371,"threads_started":1}
I20260927 23:09:01.704717 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:01.707243 25124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70095832a550474199bca5e48e3c0309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 23:09:01.707584 25124 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:01.707690 25124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70095832a550474199bca5e48e3c0309, State: Initialized, Role: FOLLOWER
I20260927 23:09:01.708297 25124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70095832a550474199bca5e48e3c0309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 23:09:01.708541 25124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:01.708678 25124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:01.708882 25124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:01.712298 25124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70095832a550474199bca5e48e3c0309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 23:09:01.712837 25124 leader_election.cc:302] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70095832a550474199bca5e48e3c0309; no voters:
I20260927 23:09:01.713653 25124 leader_election.cc:288] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:01.713675 25134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:01.716254 25124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:01.716785 25134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [term 1 LEADER]: Becoming Leader. State: Replica: 70095832a550474199bca5e48e3c0309, State: Running, Role: LEADER
I20260927 23:09:01.717303 25134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70095832a550474199bca5e48e3c0309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } }
I20260927 23:09:01.719975 25135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70095832a550474199bca5e48e3c0309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70095832a550474199bca5e48e3c0309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260927 23:09:01.720418 25135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:01.719626 25136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70095832a550474199bca5e48e3c0309. Latest consensus state: current_term: 1 leader_uuid: "70095832a550474199bca5e48e3c0309" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70095832a550474199bca5e48e3c0309" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36921 } } }
I20260927 23:09:01.721194 25136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:01.730175 25127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:01.709098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:52349 (local address 127.0.0.1:36921)
0927 23:09:01.709324 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:01.709333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:01.709728 (+ 395us) server_negotiation.cc:407] Connection header received
0927 23:09:01.709856 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:01.709860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:01.709919 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:01.710031 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:01.711868 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:01.712667 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:01.714812 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:01.715550 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:01.715663 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:01.719421 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:01.719528 (+ 107us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:01.719535 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:01.719582 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:01.724560 (+ 4978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:01.725818 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:01.725826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:01.725833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:01.726053 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:01.726826 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:01.726833 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:01.726839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:01.727414 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:01.729053 (+ 1639us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:01.729294 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 23:09:01.729547 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:09:01.736078 25143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:01.741173 25143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:01.748729 25143 catalog_manager.cc:1347] Generated new cluster ID: caadf0d211b84698af043829b0c8fdd3
I20260927 23:09:01.748826 25143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:01.761796 25143 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:01.763561 25143 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:01.772552 25143 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70095832a550474199bca5e48e3c0309: Generated new TSK 0
I20260927 23:09:01.773578 25143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:01.779366 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25057
I20260927 23:09:01.780448 25119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:01.867475 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25057
W20260927 23:09:01.884006 25042 connection.cc:535] client connection to 127.0.0.1:36921 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36921): Connection reset by peer (error 104)
2026-09-27T23:09:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:01 dist-test-slave-lmfp krb5kdc[25156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:01 dist-test-slave-lmfp krb5kdc[25156](info): set up 2 sockets
Sep 27 23:09:01 dist-test-slave-lmfp krb5kdc[25156](info): commencing operation
krb5kdc: starting...
W20260927 23:09:02.939337 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:03 dist-test-slave-lmfp krb5kdc[25156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:09:03.170153 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:03.487439 25169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:03.487721 25169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:03.487892 25169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:03.494833 25169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:03.494951 25169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:03.495085 25169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:03.495191 25169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:03.495276 25169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:03.504848 25169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:03.506124 25169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:03.507526 25169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:03.519160 25174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:03.519269 25175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:03.519239 25177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:03.519945 25169 server_base.cc:1034] running on GCE node
I20260927 23:09:03.520764 25169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:03.522148 25169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:03.523413 25169 hybrid_clock.cc:648] HybridClock initialized: now 1790550543523362 us; error 63 us; skew 500 ppm
I20260927 23:09:03.523777 25169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:03 dist-test-slave-lmfp krb5kdc[25156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:03.526655 25169 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:09:03.528071 25169 webserver.cc:460] Webserver started at http://127.0.0.1:40887/ using document root <none> and password file <none>
I20260927 23:09:03.528710 25169 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:03.528844 25169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:03.529119 25169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:03.531919 25169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "333edd3f80dd412ea58208f080fec39f"
format_stamp: "Formatted at 2026-09-27 23:09:03 on dist-test-slave-lmfp"
server_key: "ce5af1a9675a35082e5a7cff2a625921"
server_key_iv: "af66d3e992b860eff4b4b10d33ad25b6"
server_key_version: "encryptionkey@0"
I20260927 23:09:03.532635 25169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "333edd3f80dd412ea58208f080fec39f"
format_stamp: "Formatted at 2026-09-27 23:09:03 on dist-test-slave-lmfp"
server_key: "ce5af1a9675a35082e5a7cff2a625921"
server_key_iv: "af66d3e992b860eff4b4b10d33ad25b6"
server_key_version: "encryptionkey@0"
I20260927 23:09:03.537566 25169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:09:03.540886 25184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:03.542496 25169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:09:03.542675 25169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "333edd3f80dd412ea58208f080fec39f"
format_stamp: "Formatted at 2026-09-27 23:09:03 on dist-test-slave-lmfp"
server_key: "ce5af1a9675a35082e5a7cff2a625921"
server_key_iv: "af66d3e992b860eff4b4b10d33ad25b6"
server_key_version: "encryptionkey@0"
I20260927 23:09:03.542867 25169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:03.563687 25169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:03.567682 25169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:03.567958 25169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:03.585465 25169 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34327
I20260927 23:09:03.585497 25235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34327 every 8 connection(s)
I20260927 23:09:03.586735 25169 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:03.590214 25236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:03.590727 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25169
I20260927 23:09:03.590900 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:03.591341 4035 external_mini_cluster.cc:1428] Setting key e470db834d701f22047056d50048730b
I20260927 23:09:03.603514 25236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f: Bootstrap starting.
I20260927 23:09:03.605078 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:03.593111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:49885 (local address 127.0.0.1:34327)
0927 23:09:03.593898 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:03.593916 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:03.593966 (+ 50us) server_negotiation.cc:407] Connection header received
0927 23:09:03.596449 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:03.596471 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:03.596857 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:03.597163 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:03.597727 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:03.598931 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:03.600855 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:03.601491 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:03.601610 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:03.602869 (+ 1259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:03.602896 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:03.602919 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:03.603494 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:03.603566 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:03.603588 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:03.603742 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:03.604008 (+ 266us) server_negotiation.cc:299] Negotiation successful
0927 23:09:03.604191 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":285,"threads_started":1}
I20260927 23:09:03.606803 25236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:03.608057 25236 log.cc:824] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:03.608830 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:03.611143 25236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f: No bootstrap required, opened a new log
I20260927 23:09:03.620357 25239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:03.613910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:40953 (local address 127.0.0.1:34327)
0927 23:09:03.614088 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:03.614093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:03.614430 (+ 337us) server_negotiation.cc:407] Connection header received
0927 23:09:03.614580 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:03.614590 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:03.614662 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:03.614765 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:03.615324 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:03.616437 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:03.617664 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:03.618353 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:03.618485 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:03.619619 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:03.619626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:03.619631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:03.619651 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:03.619694 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:03.619713 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:03.619845 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:03.619995 (+ 150us) server_negotiation.cc:299] Negotiation successful
0927 23:09:03.620044 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:09:03.620877 25236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333edd3f80dd412ea58208f080fec39f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260927 23:09:03.621263 25236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:03.621392 25236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333edd3f80dd412ea58208f080fec39f, State: Initialized, Role: FOLLOWER
I20260927 23:09:03.621934 25236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333edd3f80dd412ea58208f080fec39f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260927 23:09:03.622167 25236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:03.622309 25236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:03.622473 25236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:03.624524 25236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333edd3f80dd412ea58208f080fec39f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260927 23:09:03.625093 25236 leader_election.cc:302] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 333edd3f80dd412ea58208f080fec39f; no voters:
I20260927 23:09:03.625624 25236 leader_election.cc:288] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:03.625765 25246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:03.626516 25246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [term 1 LEADER]: Becoming Leader. State: Replica: 333edd3f80dd412ea58208f080fec39f, State: Running, Role: LEADER
I20260927 23:09:03.627076 25246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333edd3f80dd412ea58208f080fec39f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } }
I20260927 23:09:03.627624 25236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:03.630079 25247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "333edd3f80dd412ea58208f080fec39f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333edd3f80dd412ea58208f080fec39f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } } }
I20260927 23:09:03.630501 25247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:03.630052 25248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 333edd3f80dd412ea58208f080fec39f. Latest consensus state: current_term: 1 leader_uuid: "333edd3f80dd412ea58208f080fec39f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "333edd3f80dd412ea58208f080fec39f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34327 } } }
I20260927 23:09:03.631067 25248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:03.632135 25252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:03.637022 25252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:03.646268 25252 catalog_manager.cc:1347] Generated new cluster ID: 14c5daf6e0f749038c8c13f2efdd0135
I20260927 23:09:03.646397 25252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:03.655761 25252 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:03.656949 25252 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:03.671559 25252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 333edd3f80dd412ea58208f080fec39f: Generated new TSK 0
I20260927 23:09:03.672698 25252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:03.687774 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25169
W20260927 23:09:03.688958 25231 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.85:49885
W20260927 23:09:03.689652 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25169: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:09:03.689798 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25169
W20260927 23:09:03.707880 25154 connection.cc:535] client connection to 127.0.0.1:34327 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34327): Connection reset by peer (error 104)
2026-09-27T23:09:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1823 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:03 dist-test-slave-lmfp krb5kdc[25267](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:03 dist-test-slave-lmfp krb5kdc[25267](info): set up 2 sockets
Sep 27 23:09:03 dist-test-slave-lmfp krb5kdc[25267](info): commencing operation
krb5kdc: starting...
W20260927 23:09:04.759476 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:04 dist-test-slave-lmfp krb5kdc[25267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:09:04.988188 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:05.303870 25280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:05.304145 25280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:05.304338 25280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:05.311244 25280 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:05.311336 25280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:05.311446 25280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:05.311534 25280 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:05.311597 25280 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:05.321004 25280 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:05.322279 25280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:05.323570 25280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:05.335126 25285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:05.335216 25286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:05.335177 25288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:05.336159 25280 server_base.cc:1034] running on GCE node
I20260927 23:09:05.336941 25280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:05.338270 25280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:05.339495 25280 hybrid_clock.cc:648] HybridClock initialized: now 1790550545339464 us; error 44 us; skew 500 ppm
I20260927 23:09:05.339839 25280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:05 dist-test-slave-lmfp krb5kdc[25267](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:05.345944 25280 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:09:05.347304 25280 webserver.cc:460] Webserver started at http://127.0.0.1:34283/ using document root <none> and password file <none>
I20260927 23:09:05.347946 25280 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:05.348069 25280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:05.348352 25280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:05.351090 25280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "70e476eb0fd14692ac81ac8a5b3f5713"
format_stamp: "Formatted at 2026-09-27 23:09:05 on dist-test-slave-lmfp"
server_key: "ec3d3b966dabfd3770088dd74c337c4b"
server_key_iv: "da44979cc5a03545c19c37054d4ae0c9"
server_key_version: "encryptionkey@0"
I20260927 23:09:05.351837 25280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "70e476eb0fd14692ac81ac8a5b3f5713"
format_stamp: "Formatted at 2026-09-27 23:09:05 on dist-test-slave-lmfp"
server_key: "ec3d3b966dabfd3770088dd74c337c4b"
server_key_iv: "da44979cc5a03545c19c37054d4ae0c9"
server_key_version: "encryptionkey@0"
I20260927 23:09:05.356431 25280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260927 23:09:05.359643 25295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:05.360925 25280 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:09:05.361137 25280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "70e476eb0fd14692ac81ac8a5b3f5713"
format_stamp: "Formatted at 2026-09-27 23:09:05 on dist-test-slave-lmfp"
server_key: "ec3d3b966dabfd3770088dd74c337c4b"
server_key_iv: "da44979cc5a03545c19c37054d4ae0c9"
server_key_version: "encryptionkey@0"
I20260927 23:09:05.361349 25280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:05.366977 25280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:05.370510 25280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:05.370779 25280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:05.387323 25280 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36831
I20260927 23:09:05.387367 25346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36831 every 8 connection(s)
I20260927 23:09:05.388578 25280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:05.391914 25347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:05.398054 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25280
I20260927 23:09:05.398221 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:05.398749 4035 external_mini_cluster.cc:1428] Setting key c61711bc4781d71d5a22a7fd66195661
I20260927 23:09:05.402216 25347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713: Bootstrap starting.
I20260927 23:09:05.405404 25347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:05.406203 25350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:05.400522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:42471 (local address 127.0.0.1:36831)
0927 23:09:05.401223 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:05.401234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:05.401282 (+ 48us) server_negotiation.cc:407] Connection header received
0927 23:09:05.402955 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:05.402974 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:05.403337 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:05.403714 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:05.404678 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:05.404684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:05.404697 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:05.404710 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:05.404764 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:05.404783 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:05.404983 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:05.405113 (+ 130us) server_negotiation.cc:299] Negotiation successful
0927 23:09:05.405322 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":293,"threads_started":1}
I20260927 23:09:05.406508 25347 log.cc:824] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:05.409409 25347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713: No bootstrap required, opened a new log
I20260927 23:09:05.409775 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:05.416253 25350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:05.413915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:45567 (local address 127.0.0.1:36831)
0927 23:09:05.414079 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:05.414086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:05.414495 (+ 409us) server_negotiation.cc:407] Connection header received
0927 23:09:05.414605 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:05.414610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:05.414663 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:05.414770 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:05.415643 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:05.415648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:05.415653 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:05.415669 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:05.415717 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:05.415741 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:05.415855 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:05.415937 (+ 82us) server_negotiation.cc:299] Negotiation successful
0927 23:09:05.415988 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:09:05.417201 25347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e476eb0fd14692ac81ac8a5b3f5713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36831 } }
I20260927 23:09:05.417475 25347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:05.417537 25347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e476eb0fd14692ac81ac8a5b3f5713, State: Initialized, Role: FOLLOWER
I20260927 23:09:05.417995 25347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e476eb0fd14692ac81ac8a5b3f5713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36831 } }
I20260927 23:09:05.418187 25347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:05.418277 25347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:05.418397 25347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:05.420800 25347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e476eb0fd14692ac81ac8a5b3f5713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36831 } }
I20260927 23:09:05.421190 25347 leader_election.cc:302] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e476eb0fd14692ac81ac8a5b3f5713; no voters:
I20260927 23:09:05.421666 25347 leader_election.cc:288] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:05.421780 25357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:05.422426 25357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [term 1 LEADER]: Becoming Leader. State: Replica: 70e476eb0fd14692ac81ac8a5b3f5713, State: Running, Role: LEADER
I20260927 23:09:05.422940 25357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e476eb0fd14692ac81ac8a5b3f5713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36831 } }
I20260927 23:09:05.423755 25347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:05.425952 25359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e476eb0fd14692ac81ac8a5b3f5713. Latest consensus state: current_term: 1 leader_uuid: "70e476eb0fd14692ac81ac8a5b3f5713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e476eb0fd14692ac81ac8a5b3f5713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36831 } } }
I20260927 23:09:05.426441 25358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e476eb0fd14692ac81ac8a5b3f5713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e476eb0fd14692ac81ac8a5b3f5713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36831 } } }
I20260927 23:09:05.426745 25359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:05.426798 25358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:05.431484 25367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:05.435721 25367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:05.443365 25367 catalog_manager.cc:1347] Generated new cluster ID: 94544e232ba94bf2a64315410b05e965
I20260927 23:09:05.443475 25367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:05.460629 25367 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:05.461112 25367 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:05.471469 25367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e476eb0fd14692ac81ac8a5b3f5713: Generated new TSK 0
I20260927 23:09:05.472394 25367 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:05.478962 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25280
W20260927 23:09:05.480135 25342 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.85:42471
W20260927 23:09:05.480777 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25280: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:09:05.480921 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25280
2026-09-27T23:09:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1788 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:05 dist-test-slave-lmfp krb5kdc[25378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:05 dist-test-slave-lmfp krb5kdc[25378](info): set up 2 sockets
Sep 27 23:09:05 dist-test-slave-lmfp krb5kdc[25378](info): commencing operation
krb5kdc: starting...
W20260927 23:09:06.551549 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:06 dist-test-slave-lmfp krb5kdc[25378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:09:06.792234 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:07.106600 25391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:07.107043 25391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:07.107223 25391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:07.115553 25391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:07.115667 25391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:07.115760 25391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:07.115851 25391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:07.115909 25391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:07.125236 25391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37193
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:07.126458 25391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:07.127768 25391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:07.138005 25396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:07.138926 25397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:07.139834 25399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:07.140128 25391 server_base.cc:1034] running on GCE node
I20260927 23:09:07.140899 25391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:07.142264 25391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:07.143481 25391 hybrid_clock.cc:648] HybridClock initialized: now 1790550547143447 us; error 46 us; skew 500 ppm
I20260927 23:09:07.143800 25391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:07 dist-test-slave-lmfp krb5kdc[25378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:07.146643 25391 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:09:07.148046 25391 webserver.cc:460] Webserver started at http://127.0.0.1:40901/ using document root <none> and password file <none>
I20260927 23:09:07.148702 25391 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:07.148854 25391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:07.149160 25391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:07.151993 25391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "389d0741f0b34b6c81fb01d33379611b"
format_stamp: "Formatted at 2026-09-27 23:09:07 on dist-test-slave-lmfp"
server_key: "8cfa244d4196f4ab4774e0e71887c294"
server_key_iv: "f9e6d0782f56e017020330440ab4ea55"
server_key_version: "encryptionkey@0"
I20260927 23:09:07.152701 25391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "389d0741f0b34b6c81fb01d33379611b"
format_stamp: "Formatted at 2026-09-27 23:09:07 on dist-test-slave-lmfp"
server_key: "8cfa244d4196f4ab4774e0e71887c294"
server_key_iv: "f9e6d0782f56e017020330440ab4ea55"
server_key_version: "encryptionkey@0"
I20260927 23:09:07.157581 25391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260927 23:09:07.160825 25406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:07.162127 25391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260927 23:09:07.162305 25391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "389d0741f0b34b6c81fb01d33379611b"
format_stamp: "Formatted at 2026-09-27 23:09:07 on dist-test-slave-lmfp"
server_key: "8cfa244d4196f4ab4774e0e71887c294"
server_key_iv: "f9e6d0782f56e017020330440ab4ea55"
server_key_version: "encryptionkey@0"
I20260927 23:09:07.162489 25391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:07.183542 25391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:07.187282 25391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:07.187546 25391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:07.205536 25391 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37193
I20260927 23:09:07.205590 25457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37193 every 8 connection(s)
I20260927 23:09:07.206866 25391 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:07.210304 25458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:07.214414 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25391
I20260927 23:09:07.214596 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:07.215039 4035 external_mini_cluster.cc:1428] Setting key a6d00e676bbcde816d5ecacd32ade8be
I20260927 23:09:07.220964 25458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b: Bootstrap starting.
I20260927 23:09:07.224601 25458 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:07.225724 25458 log.cc:824] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:07.228359 25458 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b: No bootstrap required, opened a new log
Sep 27 23:09:07 dist-test-slave-lmfp krb5kdc[25378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260927 23:09:07.237274 25458 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389d0741f0b34b6c81fb01d33379611b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260927 23:09:07.237619 25458 raft_consensus.cc:374] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:07.237728 25458 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389d0741f0b34b6c81fb01d33379611b, State: Initialized, Role: FOLLOWER
I20260927 23:09:07.238094 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:07.216762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:59639 (local address 127.0.0.1:37193)
0927 23:09:07.217524 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:07.217535 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:07.217574 (+ 39us) server_negotiation.cc:407] Connection header received
0927 23:09:07.219529 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:07.219550 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:07.219921 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:07.220238 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:07.221993 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:07.223353 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:07.225151 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:07.225805 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:07.226004 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:07.230611 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:07.230618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:07.230631 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:07.230672 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:07.233781 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:07.234687 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:07.234693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:07.234698 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:07.234846 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:07.235331 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:07.235334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:07.235336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:07.235843 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:07.236065 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:07.236615 (+ 550us) server_negotiation.cc:299] Negotiation successful
0927 23:09:07.236953 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":334,"threads_started":1}
I20260927 23:09:07.238381 25458 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389d0741f0b34b6c81fb01d33379611b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260927 23:09:07.238620 25458 raft_consensus.cc:388] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:07.238785 25458 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:07.238976 25458 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:07.242133 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:07.242092 25458 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389d0741f0b34b6c81fb01d33379611b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260927 23:09:07.242565 25458 leader_election.cc:302] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 389d0741f0b34b6c81fb01d33379611b; no voters:
I20260927 23:09:07.243165 25458 leader_election.cc:288] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:07.243273 25463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:07.244890 25458 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:07.247924 25463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [term 1 LEADER]: Becoming Leader. State: Replica: 389d0741f0b34b6c81fb01d33379611b, State: Running, Role: LEADER
I20260927 23:09:07.248611 25463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389d0741f0b34b6c81fb01d33379611b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } }
I20260927 23:09:07.251691 25464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "389d0741f0b34b6c81fb01d33379611b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389d0741f0b34b6c81fb01d33379611b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } } }
I20260927 23:09:07.252146 25464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:07.252735 25465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389d0741f0b34b6c81fb01d33379611b. Latest consensus state: current_term: 1 leader_uuid: "389d0741f0b34b6c81fb01d33379611b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389d0741f0b34b6c81fb01d33379611b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37193 } } }
I20260927 23:09:07.253095 25465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:07.253093 25475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:07.262286 25475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:07.272368 25486 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 23:09:07.273998 25475 catalog_manager.cc:1347] Generated new cluster ID: 121eb53fc95546dd873c89cf6b43a0fa
I20260927 23:09:07.274174 25475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:07.278769 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:07.258211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:35667 (local address 127.0.0.1:37193)
0927 23:09:07.258893 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:07.258899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:07.259364 (+ 465us) server_negotiation.cc:407] Connection header received
0927 23:09:07.259684 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:07.259693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:07.259787 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:07.259876 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:07.262524 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:07.264168 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:07.267101 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:07.268020 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:07.268164 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:07.272436 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:07.272441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:07.272449 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:07.272486 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:07.276121 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:07.276817 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:07.276822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:07.276827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:07.276949 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:07.277392 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:07.277395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:07.277397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:07.277676 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:07.277929 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:07.278354 (+ 425us) server_negotiation.cc:299] Negotiation successful
0927 23:09:07.278509 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540}
I20260927 23:09:07.297401 25475 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:07.298517 25475 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:07.306974 25475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389d0741f0b34b6c81fb01d33379611b: Generated new TSK 0
I20260927 23:09:07.307747 25475 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:07.319257 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25391
I20260927 23:09:07.320297 25453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:07.409718 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25391
W20260927 23:09:07.426362 25376 connection.cc:535] client connection to 127.0.0.1:37193 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37193): Connection reset by peer (error 104)
2026-09-27T23:09:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:07 dist-test-slave-lmfp krb5kdc[25491](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:07 dist-test-slave-lmfp krb5kdc[25491](info): set up 2 sockets
Sep 27 23:09:07 dist-test-slave-lmfp krb5kdc[25491](info): commencing operation
krb5kdc: starting...
W20260927 23:09:08.487581 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:08 dist-test-slave-lmfp krb5kdc[25491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550548, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:09:08.722014 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42417
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42417
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:09.035048 25504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:09.035310 25504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:09.035462 25504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:09.041965 25504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:09.042055 25504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:09.042156 25504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:09.042255 25504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:09.042330 25504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:09.051476 25504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42417
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:09.052714 25504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:09.053957 25504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:09.065689 25509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:09.065666 25510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:09.066051 25512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:09.066552 25504 server_base.cc:1034] running on GCE node
I20260927 23:09:09.067397 25504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:09.068776 25504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:09.070013 25504 hybrid_clock.cc:648] HybridClock initialized: now 1790550549069974 us; error 52 us; skew 500 ppm
I20260927 23:09:09.070348 25504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:09 dist-test-slave-lmfp krb5kdc[25491](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:09.073721 25504 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:09:09.075050 25504 webserver.cc:460] Webserver started at http://127.0.0.1:45773/ using document root <none> and password file <none>
I20260927 23:09:09.075783 25504 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:09.075891 25504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:09.076124 25504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:09.078912 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "31bcfdb9f53b4ce6994766d8367a1f97"
format_stamp: "Formatted at 2026-09-27 23:09:09 on dist-test-slave-lmfp"
server_key: "f1a331fd2cd3bdc7f44e1252e53ac9db"
server_key_iv: "a7c90803f96c44092b4e968a178b9fe1"
server_key_version: "encryptionkey@0"
I20260927 23:09:09.079655 25504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "31bcfdb9f53b4ce6994766d8367a1f97"
format_stamp: "Formatted at 2026-09-27 23:09:09 on dist-test-slave-lmfp"
server_key: "f1a331fd2cd3bdc7f44e1252e53ac9db"
server_key_iv: "a7c90803f96c44092b4e968a178b9fe1"
server_key_version: "encryptionkey@0"
I20260927 23:09:09.084409 25504 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260927 23:09:09.087749 25519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:09.089041 25504 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:09:09.089233 25504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "31bcfdb9f53b4ce6994766d8367a1f97"
format_stamp: "Formatted at 2026-09-27 23:09:09 on dist-test-slave-lmfp"
server_key: "f1a331fd2cd3bdc7f44e1252e53ac9db"
server_key_iv: "a7c90803f96c44092b4e968a178b9fe1"
server_key_version: "encryptionkey@0"
I20260927 23:09:09.089430 25504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:09.118535 25504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:09.122864 25504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:09.123201 25504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:09.141229 25504 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42417
I20260927 23:09:09.141250 25570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42417 every 8 connection(s)
I20260927 23:09:09.142557 25504 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:09.143885 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25504
I20260927 23:09:09.144110 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:09.144726 4035 external_mini_cluster.cc:1428] Setting key db891bd706f997edde643878cf10e3f1
I20260927 23:09:09.147778 25571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:09.157248 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:09.146813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:49161 (local address 127.0.0.1:42417)
0927 23:09:09.147570 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:09.147581 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:09.147619 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:09:09.149239 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:09.149260 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:09.149616 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:09.149990 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:09.150500 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:09.151788 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:09.153099 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:09.153740 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:09.153932 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:09.155152 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:09.155158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:09.155170 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:09.155638 (+ 468us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:09.155731 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:09.155755 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:09.155925 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:09.156196 (+ 271us) server_negotiation.cc:299] Negotiation successful
0927 23:09:09.156332 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":366,"threads_started":1}
I20260927 23:09:09.160607 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97: Bootstrap starting.
I20260927 23:09:09.161187 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:09.163904 25571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:09.165021 25571 log.cc:824] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:09.168427 25571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97: No bootstrap required, opened a new log
I20260927 23:09:09.172665 25574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:09.165610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:57943 (local address 127.0.0.1:42417)
0927 23:09:09.165834 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:09.165840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:09.166270 (+ 430us) server_negotiation.cc:407] Connection header received
0927 23:09:09.166460 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:09.166464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:09.166514 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:09.166592 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:09.167298 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:09.167991 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:09.170011 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:09.170498 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:09.170594 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:09.171851 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:09.171855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:09.171858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:09.171902 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:09.171957 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:09.171974 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:09.172054 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:09.172286 (+ 232us) server_negotiation.cc:299] Negotiation successful
0927 23:09:09.172320 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:09:09.177662 25571 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcfdb9f53b4ce6994766d8367a1f97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20260927 23:09:09.178051 25571 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:09.178161 25571 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bcfdb9f53b4ce6994766d8367a1f97, State: Initialized, Role: FOLLOWER
I20260927 23:09:09.178761 25571 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcfdb9f53b4ce6994766d8367a1f97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20260927 23:09:09.179062 25571 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:09.179212 25571 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:09.179397 25571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:09.181622 25571 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcfdb9f53b4ce6994766d8367a1f97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20260927 23:09:09.182117 25571 leader_election.cc:302] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31bcfdb9f53b4ce6994766d8367a1f97; no voters:
I20260927 23:09:09.182703 25571 leader_election.cc:288] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:09.182793 25581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:09.183642 25581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [term 1 LEADER]: Becoming Leader. State: Replica: 31bcfdb9f53b4ce6994766d8367a1f97, State: Running, Role: LEADER
I20260927 23:09:09.184134 25581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcfdb9f53b4ce6994766d8367a1f97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } }
I20260927 23:09:09.185142 25571 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:09.186879 25582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31bcfdb9f53b4ce6994766d8367a1f97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcfdb9f53b4ce6994766d8367a1f97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } } }
I20260927 23:09:09.187060 25583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31bcfdb9f53b4ce6994766d8367a1f97. Latest consensus state: current_term: 1 leader_uuid: "31bcfdb9f53b4ce6994766d8367a1f97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31bcfdb9f53b4ce6994766d8367a1f97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42417 } } }
I20260927 23:09:09.188351 25582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:09.188351 25583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:09.189203 25588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:09.193745 25588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:09.204957 25588 catalog_manager.cc:1347] Generated new cluster ID: cc7ff0f0b3f54cf0915af024086df8e5
I20260927 23:09:09.205060 25588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:09.214939 25588 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:09.216342 25588 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:09.227586 25588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31bcfdb9f53b4ce6994766d8367a1f97: Generated new TSK 0
I20260927 23:09:09.228294 25588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:09.242623 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25504
W20260927 23:09:09.244016 25566 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.85:49161
W20260927 23:09:09.244691 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25504: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:09:09.244812 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25504
W20260927 23:09:09.260988 25489 connection.cc:535] client connection to 127.0.0.1:42417 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42417): Connection reset by peer (error 104)
2026-09-27T23:09:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1838 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:09 dist-test-slave-lmfp krb5kdc[25602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:09 dist-test-slave-lmfp krb5kdc[25602](info): set up 2 sockets
Sep 27 23:09:09 dist-test-slave-lmfp krb5kdc[25602](info): commencing operation
krb5kdc: starting...
W20260927 23:09:10.315299 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:10 dist-test-slave-lmfp krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260927 23:09:10.554464 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:10.880471 25615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:10.880805 25615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:10.880993 25615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:10.888130 25615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:10.888274 25615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:10.888382 25615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:10.888499 25615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:10.888588 25615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:10.897980 25615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:10.899259 25615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:10.900547 25615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:10.912536 25620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:10.912582 25621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:10.912639 25623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:10.913439 25615 server_base.cc:1034] running on GCE node
I20260927 23:09:10.914181 25615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:10.915561 25615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:10.916760 25615 hybrid_clock.cc:648] HybridClock initialized: now 1790550550916688 us; error 68 us; skew 500 ppm
I20260927 23:09:10.917097 25615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:10 dist-test-slave-lmfp krb5kdc[25602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:10.923738 25615 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260927 23:09:10.925102 25615 webserver.cc:460] Webserver started at http://127.0.0.1:41751/ using document root <none> and password file <none>
I20260927 23:09:10.925750 25615 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:10.925846 25615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:10.926090 25615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:10.928932 25615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b"
format_stamp: "Formatted at 2026-09-27 23:09:10 on dist-test-slave-lmfp"
server_key: "453cd0627d8e88fbd206deed3a88ec6a"
server_key_iv: "40e4d7d27816c7ce7881cc392accce54"
server_key_version: "encryptionkey@0"
I20260927 23:09:10.929668 25615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b"
format_stamp: "Formatted at 2026-09-27 23:09:10 on dist-test-slave-lmfp"
server_key: "453cd0627d8e88fbd206deed3a88ec6a"
server_key_iv: "40e4d7d27816c7ce7881cc392accce54"
server_key_version: "encryptionkey@0"
I20260927 23:09:10.934512 25615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260927 23:09:10.937901 25630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:10.939188 25615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:09:10.939352 25615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b"
format_stamp: "Formatted at 2026-09-27 23:09:10 on dist-test-slave-lmfp"
server_key: "453cd0627d8e88fbd206deed3a88ec6a"
server_key_iv: "40e4d7d27816c7ce7881cc392accce54"
server_key_version: "encryptionkey@0"
I20260927 23:09:10.939532 25615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:10.944757 25615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:10.948467 25615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:10.948733 25615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:10.966068 25615 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36149
I20260927 23:09:10.966089 25681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36149 every 8 connection(s)
I20260927 23:09:10.967370 25615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:10.970536 25682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:10.975692 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25615
I20260927 23:09:10.975858 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:10.976341 4035 external_mini_cluster.cc:1428] Setting key 6f16fa4857a4a2d1f82cf4c710a2c640
I20260927 23:09:10.981124 25682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b: Bootstrap starting.
I20260927 23:09:10.983551 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:10.977965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:54043 (local address 127.0.0.1:36149)
0927 23:09:10.978662 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:10.978671 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:10.978722 (+ 51us) server_negotiation.cc:407] Connection header received
0927 23:09:10.980477 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:10.980498 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:10.980790 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:10.981117 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:10.982100 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:10.982105 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:10.982117 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:10.982135 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:10.982181 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:10.982198 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:10.982350 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:10.982460 (+ 110us) server_negotiation.cc:299] Negotiation successful
0927 23:09:10.982642 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":325,"threads_started":1}
I20260927 23:09:10.984412 25682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:10.985390 25682 log.cc:824] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:10.986855 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:10.988376 25682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b: No bootstrap required, opened a new log
I20260927 23:09:10.993896 25685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:10.991511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.85:41435 (local address 127.0.0.1:36149)
0927 23:09:10.991679 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:10.991685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:10.992091 (+ 406us) server_negotiation.cc:407] Connection header received
0927 23:09:10.992195 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:10.992204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:10.992272 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:10.992390 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:10.993267 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:10.993272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:10.993277 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:10.993297 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:10.993345 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:10.993364 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:10.993511 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:10.993560 (+ 49us) server_negotiation.cc:299] Negotiation successful
0927 23:09:10.993619 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 23:09:10.997040 25682 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } }
I20260927 23:09:10.997350 25682 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:10.997416 25682 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e0ca0e1ddd4b7d8aee9332a643eb3b, State: Initialized, Role: FOLLOWER
I20260927 23:09:10.997840 25682 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } }
I20260927 23:09:10.998049 25682 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:10.998134 25682 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:10.998247 25682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:11.000418 25682 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } }
I20260927 23:09:11.000923 25682 leader_election.cc:302] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99e0ca0e1ddd4b7d8aee9332a643eb3b; no voters:
I20260927 23:09:11.001523 25682 leader_election.cc:288] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:11.001681 25692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:11.002312 25692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [term 1 LEADER]: Becoming Leader. State: Replica: 99e0ca0e1ddd4b7d8aee9332a643eb3b, State: Running, Role: LEADER
I20260927 23:09:11.002760 25692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } }
I20260927 23:09:11.003509 25682 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:11.005215 25693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } } }
I20260927 23:09:11.005581 25693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:11.006059 25694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99e0ca0e1ddd4b7d8aee9332a643eb3b. Latest consensus state: current_term: 1 leader_uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99e0ca0e1ddd4b7d8aee9332a643eb3b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36149 } } }
I20260927 23:09:11.006335 25694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:11.007097 25697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:11.014292 25697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:11.022591 25697 catalog_manager.cc:1347] Generated new cluster ID: f5786f337b394c229ce92dc981144c29
I20260927 23:09:11.022715 25697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:11.045755 25697 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:11.046272 25697 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:11.056282 25697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99e0ca0e1ddd4b7d8aee9332a643eb3b: Generated new TSK 0
I20260927 23:09:11.057037 25697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:11.074426 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25615
W20260927 23:09:11.075666 25677 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.85:54043
W20260927 23:09:11.076329 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25615: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:09:11.076484 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25615
2026-09-27T23:09:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1827 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:11 dist-test-slave-lmfp krb5kdc[25713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:11 dist-test-slave-lmfp krb5kdc[25713](info): set up 2 sockets
Sep 27 23:09:11 dist-test-slave-lmfp krb5kdc[25713](info): commencing operation
krb5kdc: starting...
W20260927 23:09:12.159488 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:12 dist-test-slave-lmfp krb5kdc[25713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:09:12.383715 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38845
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:12.695063 25726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:12.695380 25726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:12.695560 25726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:12.702419 25726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:12.702564 25726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:12.702670 25726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:12.702762 25726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:12.702827 25726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:12.712118 25726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38845
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:12.713303 25726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:12.714550 25726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:12.725714 25731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:12.725802 25734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:12.725762 25732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:12.726882 25726 server_base.cc:1034] running on GCE node
I20260927 23:09:12.727682 25726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:12.728955 25726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:12.730183 25726 hybrid_clock.cc:648] HybridClock initialized: now 1790550552730155 us; error 57 us; skew 500 ppm
I20260927 23:09:12.730508 25726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:12 dist-test-slave-lmfp krb5kdc[25713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550552, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:12.733505 25726 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:09:12.735060 25726 webserver.cc:460] Webserver started at http://127.3.240.254:36645/ using document root <none> and password file <none>
I20260927 23:09:12.735705 25726 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:12.735854 25726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:12.736157 25726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:12.739084 25726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "12b20f4939644a7fa60e65e091a696c9"
format_stamp: "Formatted at 2026-09-27 23:09:12 on dist-test-slave-lmfp"
server_key: "517f8dcfee9873564241767effb74dcf"
server_key_iv: "96f4689c7080e2aefe86e2dc8d275d72"
server_key_version: "encryptionkey@0"
I20260927 23:09:12.739853 25726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "12b20f4939644a7fa60e65e091a696c9"
format_stamp: "Formatted at 2026-09-27 23:09:12 on dist-test-slave-lmfp"
server_key: "517f8dcfee9873564241767effb74dcf"
server_key_iv: "96f4689c7080e2aefe86e2dc8d275d72"
server_key_version: "encryptionkey@0"
I20260927 23:09:12.744777 25726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260927 23:09:12.749039 25741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:12.750206 25726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:09:12.750406 25726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "12b20f4939644a7fa60e65e091a696c9"
format_stamp: "Formatted at 2026-09-27 23:09:12 on dist-test-slave-lmfp"
server_key: "517f8dcfee9873564241767effb74dcf"
server_key_iv: "96f4689c7080e2aefe86e2dc8d275d72"
server_key_version: "encryptionkey@0"
I20260927 23:09:12.750597 25726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:12.779567 25726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:12.783895 25726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:12.784225 25726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:12.804181 25726 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:38845
I20260927 23:09:12.804199 25792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:38845 every 8 connection(s)
I20260927 23:09:12.805584 25726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:12.809245 25793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:12.812462 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25726
I20260927 23:09:12.812636 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:12.813236 4035 external_mini_cluster.cc:1428] Setting key 7b55a7e5c4b2597c686b5c54d59d67e5
I20260927 23:09:12.821729 25793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9: Bootstrap starting.
I20260927 23:09:12.825464 25793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:12.826753 25793 log.cc:824] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9: Log is configured to *not* fsync() on all Append() calls
Sep 27 23:09:12 dist-test-slave-lmfp krb5kdc[25713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550552, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:09:12.829720 25793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9: No bootstrap required, opened a new log
I20260927 23:09:12.837668 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:12.815098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48517 (local address 127.3.240.254:38845)
0927 23:09:12.815899 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:12.815910 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:12.815952 (+ 42us) server_negotiation.cc:407] Connection header received
0927 23:09:12.817714 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:12.817736 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:12.818155 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:12.818489 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:12.820409 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:12.821727 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:12.823553 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:12.824210 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:12.824335 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:12.829104 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:12.829118 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:12.829138 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:12.829179 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:12.833167 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:12.833973 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:12.833982 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:12.833989 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:12.834207 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:12.834610 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:12.834615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:12.834618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:12.835359 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:12.835689 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:12.835995 (+ 306us) server_negotiation.cc:299] Negotiation successful
0927 23:09:12.836332 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":393,"threads_started":1}
I20260927 23:09:12.840322 25793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b20f4939644a7fa60e65e091a696c9" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38845 } }
I20260927 23:09:12.840667 25793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:12.840790 25793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12b20f4939644a7fa60e65e091a696c9, State: Initialized, Role: FOLLOWER
I20260927 23:09:12.841460 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:12.841403 25793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b20f4939644a7fa60e65e091a696c9" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38845 } }
I20260927 23:09:12.841697 25793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:12.841840 25793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:12.842005 25793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:12.844096 25793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b20f4939644a7fa60e65e091a696c9" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38845 } }
I20260927 23:09:12.844558 25793 leader_election.cc:302] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12b20f4939644a7fa60e65e091a696c9; no voters:
I20260927 23:09:12.845160 25793 leader_election.cc:288] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:12.845322 25800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:12.847328 25800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [term 1 LEADER]: Becoming Leader. State: Replica: 12b20f4939644a7fa60e65e091a696c9, State: Running, Role: LEADER
I20260927 23:09:12.848032 25800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b20f4939644a7fa60e65e091a696c9" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38845 } }
I20260927 23:09:12.849906 25793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:12.851157 25802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12b20f4939644a7fa60e65e091a696c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b20f4939644a7fa60e65e091a696c9" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38845 } } }
I20260927 23:09:12.851537 25802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:12.851984 25801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12b20f4939644a7fa60e65e091a696c9. Latest consensus state: current_term: 1 leader_uuid: "12b20f4939644a7fa60e65e091a696c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12b20f4939644a7fa60e65e091a696c9" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38845 } } }
I20260927 23:09:12.852236 25801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:12.860136 25807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:12.865171 25807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:12.874953 25807 catalog_manager.cc:1347] Generated new cluster ID: 05e17708cfa0417dbcd96aba674a491f
I20260927 23:09:12.875065 25807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:12.876884 25796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:12.850231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52959 (local address 127.3.240.254:38845)
0927 23:09:12.850387 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:12.850394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:12.860279 (+ 9885us) server_negotiation.cc:407] Connection header received
0927 23:09:12.860438 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:12.860445 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:12.860542 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:12.860647 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:12.862503 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:12.863688 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:12.865372 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:12.866042 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:12.866170 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:12.869342 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:12.869349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:12.869353 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:12.869391 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:12.873061 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:12.873916 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:12.873921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:12.873924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:12.874079 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:12.874637 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:12.874642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:12.874645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:12.875095 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:12.875355 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:12.876436 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0927 23:09:12.876582 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 23:09:12.889443 25807 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:12.890553 25807 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:12.897248 25807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12b20f4939644a7fa60e65e091a696c9: Generated new TSK 0
I20260927 23:09:12.897958 25807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:12.902971 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25726
I20260927 23:09:12.904197 25788 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:12.987478 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25726
W20260927 23:09:13.003850 25711 connection.cc:535] client connection to 127.3.240.254:38845 recv error: Network error: failed to read from TLS socket (remote: 127.3.240.254:38845): Connection reset by peer (error 104)
2026-09-27T23:09:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1911 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:13 dist-test-slave-lmfp krb5kdc[25825](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:13 dist-test-slave-lmfp krb5kdc[25825](info): set up 2 sockets
Sep 27 23:09:13 dist-test-slave-lmfp krb5kdc[25825](info): commencing operation
krb5kdc: starting...
W20260927 23:09:14.059585 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:14 dist-test-slave-lmfp krb5kdc[25825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:09:14.295876 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35835
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:35835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:14.606091 25838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:14.606473 25838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:14.606635 25838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:14.613421 25838 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:14.613562 25838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:14.613662 25838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:14.613742 25838 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:14.613842 25838 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:14.623126 25838 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:35835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:14.624293 25838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:14.625506 25838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:14.637204 25844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:14.637172 25843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:14.637172 25846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:14.637977 25838 server_base.cc:1034] running on GCE node
I20260927 23:09:14.638762 25838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:14.640156 25838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:14.641376 25838 hybrid_clock.cc:648] HybridClock initialized: now 1790550554641329 us; error 47 us; skew 500 ppm
I20260927 23:09:14.641743 25838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:14 dist-test-slave-lmfp krb5kdc[25825](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550554, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:14.644734 25838 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:09:14.646164 25838 webserver.cc:460] Webserver started at http://127.3.240.254:45217/ using document root <none> and password file <none>
I20260927 23:09:14.646796 25838 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:14.646973 25838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:14.647302 25838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:14.650058 25838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "459d2d286d6a46c68fda503c236a9e2d"
format_stamp: "Formatted at 2026-09-27 23:09:14 on dist-test-slave-lmfp"
server_key: "31e431859e0b5f813bf8079e7a01180a"
server_key_iv: "aff96a1bbb4ba83d3e095c410984038e"
server_key_version: "encryptionkey@0"
I20260927 23:09:14.650797 25838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "459d2d286d6a46c68fda503c236a9e2d"
format_stamp: "Formatted at 2026-09-27 23:09:14 on dist-test-slave-lmfp"
server_key: "31e431859e0b5f813bf8079e7a01180a"
server_key_iv: "aff96a1bbb4ba83d3e095c410984038e"
server_key_version: "encryptionkey@0"
I20260927 23:09:14.655699 25838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260927 23:09:14.658932 25853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:14.660130 25838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:09:14.660279 25838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "459d2d286d6a46c68fda503c236a9e2d"
format_stamp: "Formatted at 2026-09-27 23:09:14 on dist-test-slave-lmfp"
server_key: "31e431859e0b5f813bf8079e7a01180a"
server_key_iv: "aff96a1bbb4ba83d3e095c410984038e"
server_key_version: "encryptionkey@0"
I20260927 23:09:14.660434 25838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:14.681349 25838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:14.685318 25838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:14.685604 25838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:14.703473 25838 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:35835
I20260927 23:09:14.703466 25904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:35835 every 8 connection(s)
I20260927 23:09:14.704715 25838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:14.708009 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25838
I20260927 23:09:14.708177 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:14.708300 25905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:14.708644 4035 external_mini_cluster.cc:1428] Setting key 1bce1bafb42175ab11d22db4502b3220
I20260927 23:09:14.720259 25905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d: Bootstrap starting.
I20260927 23:09:14.720762 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:14.710289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35273 (local address 127.3.240.254:35835)
0927 23:09:14.711124 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:14.711136 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:14.711180 (+ 44us) server_negotiation.cc:407] Connection header received
0927 23:09:14.712886 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:14.712906 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:14.713268 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:14.713652 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:14.714141 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:14.715394 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:14.716612 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:14.717263 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:14.717468 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:14.718635 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:14.718642 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:14.718653 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:14.719182 (+ 529us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:14.719247 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:14.719270 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:14.719423 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:14.719686 (+ 263us) server_negotiation.cc:299] Negotiation successful
0927 23:09:14.719831 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":347,"threads_started":1}
I20260927 23:09:14.723646 25905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:14.724480 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:14.724851 25905 log.cc:824] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:14.727787 25905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d: No bootstrap required, opened a new log
I20260927 23:09:14.735415 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:14.729162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56843 (local address 127.3.240.254:35835)
0927 23:09:14.729312 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:14.729318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:14.729760 (+ 442us) server_negotiation.cc:407] Connection header received
0927 23:09:14.729867 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:14.729872 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:14.729947 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:14.730040 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:14.730446 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:14.731595 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:14.732828 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:14.733548 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:14.733671 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:14.734675 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:14.734686 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:14.734691 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:14.734714 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:14.734758 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:14.734781 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:14.734885 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:14.735054 (+ 169us) server_negotiation.cc:299] Negotiation successful
0927 23:09:14.735100 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260927 23:09:14.737100 25905 raft_consensus.cc:348] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459d2d286d6a46c68fda503c236a9e2d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35835 } }
I20260927 23:09:14.737442 25905 raft_consensus.cc:374] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:14.737555 25905 raft_consensus.cc:729] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 459d2d286d6a46c68fda503c236a9e2d, State: Initialized, Role: FOLLOWER
I20260927 23:09:14.738091 25905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459d2d286d6a46c68fda503c236a9e2d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35835 } }
I20260927 23:09:14.738339 25905 raft_consensus.cc:388] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:14.738476 25905 raft_consensus.cc:482] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:14.738651 25905 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:14.740800 25905 raft_consensus.cc:504] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459d2d286d6a46c68fda503c236a9e2d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35835 } }
I20260927 23:09:14.741281 25905 leader_election.cc:302] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 459d2d286d6a46c68fda503c236a9e2d; no voters:
I20260927 23:09:14.741814 25905 leader_election.cc:288] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:14.741959 25915 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:14.742735 25915 raft_consensus.cc:686] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [term 1 LEADER]: Becoming Leader. State: Replica: 459d2d286d6a46c68fda503c236a9e2d, State: Running, Role: LEADER
I20260927 23:09:14.743252 25915 consensus_queue.cc:237] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459d2d286d6a46c68fda503c236a9e2d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35835 } }
I20260927 23:09:14.743966 25905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:14.745952 25916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "459d2d286d6a46c68fda503c236a9e2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459d2d286d6a46c68fda503c236a9e2d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35835 } } }
I20260927 23:09:14.746344 25916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:14.747159 25917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 459d2d286d6a46c68fda503c236a9e2d. Latest consensus state: current_term: 1 leader_uuid: "459d2d286d6a46c68fda503c236a9e2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "459d2d286d6a46c68fda503c236a9e2d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35835 } } }
I20260927 23:09:14.747620 25917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:14.748095 25920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:14.752421 25920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:14.760705 25920 catalog_manager.cc:1347] Generated new cluster ID: 0a2d0e5198ec4e329e88bdce89054a59
I20260927 23:09:14.760795 25920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:14.770848 25920 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:14.772555 25920 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:14.782388 25920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 459d2d286d6a46c68fda503c236a9e2d: Generated new TSK 0
I20260927 23:09:14.783286 25920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:14.793111 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25838
W20260927 23:09:14.794600 25900 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35273
W20260927 23:09:14.795326 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25838: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:09:14.795469 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25838
W20260927 23:09:14.811156 25823 connection.cc:535] client connection to 127.3.240.254:35835 recv error: Network error: failed to read from TLS socket (remote: 127.3.240.254:35835): Connection reset by peer (error 104)
2026-09-27T23:09:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1807 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:14 dist-test-slave-lmfp krb5kdc[25936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:14 dist-test-slave-lmfp krb5kdc[25936](info): set up 2 sockets
Sep 27 23:09:14 dist-test-slave-lmfp krb5kdc[25936](info): commencing operation
krb5kdc: starting...
W20260927 23:09:15.831233 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:15 dist-test-slave-lmfp krb5kdc[25936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:09:16.053579 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41443
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:41443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:16.371989 25949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:16.372354 25949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:16.372530 25949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:16.379372 25949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:16.379519 25949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:16.379652 25949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:16.379770 25949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:16.379860 25949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:16.389269 25949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:41443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:16.390612 25949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:16.391891 25949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:16.403539 25954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:16.403530 25955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:16.403705 25957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:16.404012 25949 server_base.cc:1034] running on GCE node
I20260927 23:09:16.404816 25949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:16.406183 25949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:16.407416 25949 hybrid_clock.cc:648] HybridClock initialized: now 1790550556407378 us; error 60 us; skew 500 ppm
I20260927 23:09:16.407752 25949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:16 dist-test-slave-lmfp krb5kdc[25936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550556, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:16.415417 25949 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:09:16.416785 25949 webserver.cc:460] Webserver started at http://127.3.240.254:43083/ using document root <none> and password file <none>
I20260927 23:09:16.417696 25949 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:16.417816 25949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:16.418215 25949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:16.420954 25949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52"
format_stamp: "Formatted at 2026-09-27 23:09:16 on dist-test-slave-lmfp"
server_key: "53e655a41599f7d8cc1fa50036584662"
server_key_iv: "d8faaf4f87e07f1e1c9e6611d558b5b4"
server_key_version: "encryptionkey@0"
I20260927 23:09:16.421597 25949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52"
format_stamp: "Formatted at 2026-09-27 23:09:16 on dist-test-slave-lmfp"
server_key: "53e655a41599f7d8cc1fa50036584662"
server_key_iv: "d8faaf4f87e07f1e1c9e6611d558b5b4"
server_key_version: "encryptionkey@0"
I20260927 23:09:16.426657 25949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260927 23:09:16.429721 25964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:16.430859 25949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:09:16.431041 25949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52"
format_stamp: "Formatted at 2026-09-27 23:09:16 on dist-test-slave-lmfp"
server_key: "53e655a41599f7d8cc1fa50036584662"
server_key_iv: "d8faaf4f87e07f1e1c9e6611d558b5b4"
server_key_version: "encryptionkey@0"
I20260927 23:09:16.431237 25949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:16.436308 25949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:16.439842 25949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:16.440104 25949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:16.456769 25949 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:41443
I20260927 23:09:16.456808 26015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:41443 every 8 connection(s)
I20260927 23:09:16.458082 25949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:16.461496 26016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:16.464771 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 25949
I20260927 23:09:16.464984 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:16.465404 4035 external_mini_cluster.cc:1428] Setting key 79cc7f8e3fb3ddf2e6358f2a1c726c48
I20260927 23:09:16.473610 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:16.467092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41607 (local address 127.3.240.254:41443)
0927 23:09:16.467884 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:16.467900 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:16.467948 (+ 48us) server_negotiation.cc:407] Connection header received
0927 23:09:16.469870 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:16.469901 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:16.470321 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:16.470758 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:16.471795 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:16.471803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:16.471818 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:16.471844 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:16.471909 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:16.471933 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:16.472158 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:16.472299 (+ 141us) server_negotiation.cc:299] Negotiation successful
0927 23:09:16.472515 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":331,"threads_started":1}
I20260927 23:09:16.474370 26016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52: Bootstrap starting.
I20260927 23:09:16.476897 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:16.477412 26016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:16.478339 26016 log.cc:824] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:16.481252 26016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52: No bootstrap required, opened a new log
I20260927 23:09:16.483355 26019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:16.480936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44467 (local address 127.3.240.254:41443)
0927 23:09:16.481125 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:16.481131 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:16.481483 (+ 352us) server_negotiation.cc:407] Connection header received
0927 23:09:16.481689 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:16.481694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:16.481743 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:16.481844 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:16.482724 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:16.482729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:16.482733 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:16.482752 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:16.482797 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:16.482813 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:16.482944 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:16.483068 (+ 124us) server_negotiation.cc:299] Negotiation successful
0927 23:09:16.483119 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260927 23:09:16.488864 26016 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41443 } }
I20260927 23:09:16.489209 26016 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:16.489320 26016 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8ea17b7a1a4814bd6e0fe80d695e52, State: Initialized, Role: FOLLOWER
I20260927 23:09:16.489797 26016 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41443 } }
I20260927 23:09:16.490023 26016 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:16.490156 26016 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:16.490302 26016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:16.492400 26016 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41443 } }
I20260927 23:09:16.492873 26016 leader_election.cc:302] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8ea17b7a1a4814bd6e0fe80d695e52; no voters:
I20260927 23:09:16.493444 26016 leader_election.cc:288] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:16.493584 26026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:16.495038 26026 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [term 1 LEADER]: Becoming Leader. State: Replica: fd8ea17b7a1a4814bd6e0fe80d695e52, State: Running, Role: LEADER
I20260927 23:09:16.495584 26016 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:16.495501 26026 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41443 } }
I20260927 23:09:16.498315 26028 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8ea17b7a1a4814bd6e0fe80d695e52. Latest consensus state: current_term: 1 leader_uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41443 } } }
I20260927 23:09:16.498792 26028 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:16.499356 26027 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8ea17b7a1a4814bd6e0fe80d695e52" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41443 } } }
I20260927 23:09:16.499603 26027 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52 [sys.catalog]: This master's current role is: LEADER
W20260927 23:09:16.505620 26042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:09:16.505762 26042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:09:16.505906 26035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:16.507503 26035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:16.514288 26035 catalog_manager.cc:1347] Generated new cluster ID: c47c90f60b2b4523a64fb70e82e7d2d2
I20260927 23:09:16.514380 26035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:16.537283 26035 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:16.537698 26035 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:16.544982 26035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8ea17b7a1a4814bd6e0fe80d695e52: Generated new TSK 0
I20260927 23:09:16.545724 26035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:16.550895 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25949
W20260927 23:09:16.552443 26011 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41607
W20260927 23:09:16.553051 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 25949: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:09:16.553203 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 25949
2026-09-27T23:09:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1755 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:16 dist-test-slave-lmfp krb5kdc[26047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:16 dist-test-slave-lmfp krb5kdc[26047](info): set up 2 sockets
Sep 27 23:09:16 dist-test-slave-lmfp krb5kdc[26047](info): commencing operation
krb5kdc: starting...
W20260927 23:09:17.611418 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:17 dist-test-slave-lmfp krb5kdc[26047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:09:17.833983 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39861
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:18.139331 26060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:18.139663 26060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:18.139827 26060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:18.146731 26060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:18.146875 26060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:18.146975 26060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:18.147095 26060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:18.147168 26060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:18.156201 26060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:18.157441 26060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:18.158682 26060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:18.169790 26065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:18.169742 26068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:18.169831 26066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:18.171401 26060 server_base.cc:1034] running on GCE node
I20260927 23:09:18.172078 26060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:18.173274 26060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:18.174495 26060 hybrid_clock.cc:648] HybridClock initialized: now 1790550558174458 us; error 53 us; skew 500 ppm
I20260927 23:09:18.174824 26060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:18 dist-test-slave-lmfp krb5kdc[26047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550558, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:18.177678 26060 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:09:18.179030 26060 webserver.cc:460] Webserver started at http://127.3.240.254:42673/ using document root <none> and password file <none>
I20260927 23:09:18.179631 26060 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:18.179760 26060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:18.180033 26060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:18.182788 26060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "1c4d691866f248519de5aedf41028458"
format_stamp: "Formatted at 2026-09-27 23:09:18 on dist-test-slave-lmfp"
server_key: "4acf21b859e4b166ed15a2cbda02f4ba"
server_key_iv: "3d6bb19a8cb9e4c9575f996e46ae91e6"
server_key_version: "encryptionkey@0"
I20260927 23:09:18.183588 26060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "1c4d691866f248519de5aedf41028458"
format_stamp: "Formatted at 2026-09-27 23:09:18 on dist-test-slave-lmfp"
server_key: "4acf21b859e4b166ed15a2cbda02f4ba"
server_key_iv: "3d6bb19a8cb9e4c9575f996e46ae91e6"
server_key_version: "encryptionkey@0"
I20260927 23:09:18.188398 26060 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260927 23:09:18.191859 26075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:18.193136 26060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260927 23:09:18.193306 26060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "1c4d691866f248519de5aedf41028458"
format_stamp: "Formatted at 2026-09-27 23:09:18 on dist-test-slave-lmfp"
server_key: "4acf21b859e4b166ed15a2cbda02f4ba"
server_key_iv: "3d6bb19a8cb9e4c9575f996e46ae91e6"
server_key_version: "encryptionkey@0"
I20260927 23:09:18.193480 26060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:18.217355 26060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:18.220930 26060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:18.221206 26060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:18.238926 26060 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39861
I20260927 23:09:18.238955 26126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39861 every 8 connection(s)
I20260927 23:09:18.240226 26060 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:18.243320 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26060
I20260927 23:09:18.243474 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:18.243894 4035 external_mini_cluster.cc:1428] Setting key 60e50b9273ce9b4cc73f88e1f028de90
I20260927 23:09:18.244258 26127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:18.258456 26127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458: Bootstrap starting.
Sep 27 23:09:18 dist-test-slave-lmfp krb5kdc[26047](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:09:18.262625 26127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:18.264048 26127 log.cc:824] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:18.267555 26127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458: No bootstrap required, opened a new log
I20260927 23:09:18.268805 26130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:18.245669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56675 (local address 127.3.240.254:39861)
0927 23:09:18.246504 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:18.246520 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:18.246572 (+ 52us) server_negotiation.cc:407] Connection header received
0927 23:09:18.249294 (+ 2722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:18.249320 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:18.249811 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:18.250238 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:18.251482 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:18.253377 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:18.254627 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:18.255708 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:18.255912 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:18.259961 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:18.259974 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:18.259998 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:18.260061 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:18.264229 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:18.265059 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:18.265065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:18.265070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:18.265259 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:18.265652 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:18.265657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:18.265661 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:18.266317 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:18.266680 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:18.267017 (+ 337us) server_negotiation.cc:299] Negotiation successful
0927 23:09:18.267439 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":342,"threads_started":1}
I20260927 23:09:18.272473 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:18.276674 26127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4d691866f248519de5aedf41028458" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39861 } }
I20260927 23:09:18.277137 26127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:18.277256 26127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c4d691866f248519de5aedf41028458, State: Initialized, Role: FOLLOWER
I20260927 23:09:18.277889 26127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4d691866f248519de5aedf41028458" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39861 } }
I20260927 23:09:18.278172 26127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:18.278328 26127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:18.278512 26127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:18.281237 26127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4d691866f248519de5aedf41028458" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39861 } }
I20260927 23:09:18.281780 26127 leader_election.cc:302] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c4d691866f248519de5aedf41028458; no voters:
I20260927 23:09:18.282445 26127 leader_election.cc:288] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:18.282562 26137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:18.285027 26127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:18.286926 26137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [term 1 LEADER]: Becoming Leader. State: Replica: 1c4d691866f248519de5aedf41028458, State: Running, Role: LEADER
I20260927 23:09:18.288885 26137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4d691866f248519de5aedf41028458" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39861 } }
I20260927 23:09:18.291837 26130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:18.276566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51827 (local address 127.3.240.254:39861)
0927 23:09:18.276786 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:18.276794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:18.277186 (+ 392us) server_negotiation.cc:407] Connection header received
0927 23:09:18.277391 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:18.277395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:18.277466 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:18.277575 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:18.279259 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:18.280330 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:18.281400 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:18.281856 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:18.281987 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:18.284496 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:18.284508 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:18.284512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:18.284545 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:18.288146 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:18.288855 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:18.288862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:18.288869 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:18.288985 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:18.289547 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:18.289555 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:18.289559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:18.289946 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:18.290893 (+ 947us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:18.291104 (+ 211us) server_negotiation.cc:299] Negotiation successful
0927 23:09:18.291347 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 23:09:18.291973 26139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c4d691866f248519de5aedf41028458. Latest consensus state: current_term: 1 leader_uuid: "1c4d691866f248519de5aedf41028458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4d691866f248519de5aedf41028458" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39861 } } }
I20260927 23:09:18.292399 26139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:18.293143 26138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c4d691866f248519de5aedf41028458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c4d691866f248519de5aedf41028458" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39861 } } }
I20260927 23:09:18.293483 26138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:18.295836 26151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:18.300179 26151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:18.307561 26151 catalog_manager.cc:1347] Generated new cluster ID: 3dd1334e06e84f51972323845c3c5e6c
I20260927 23:09:18.307653 26151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:18.327191 26151 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:18.328987 26151 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:18.338578 26151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c4d691866f248519de5aedf41028458: Generated new TSK 0
I20260927 23:09:18.339648 26151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:18.352461 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26060
I20260927 23:09:18.353482 26122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:18.440971 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26060
2026-09-27T23:09:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1890 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:18 dist-test-slave-lmfp krb5kdc[26160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:18 dist-test-slave-lmfp krb5kdc[26160](info): set up 2 sockets
Sep 27 23:09:18 dist-test-slave-lmfp krb5kdc[26160](info): commencing operation
krb5kdc: starting...
W20260927 23:09:19.507637 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:19 dist-test-slave-lmfp krb5kdc[26160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:09:19.731825 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45675
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:20.042026 26173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:20.042279 26173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:20.042482 26173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:20.049023 26173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:20.049144 26173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:20.049243 26173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:20.049345 26173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:20.049422 26173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:20.058465 26173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:20.059624 26173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:20.060880 26173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:20.072870 26178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:20.072835 26181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:20.072906 26179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:20.073410 26173 server_base.cc:1034] running on GCE node
I20260927 23:09:20.074203 26173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:20.075579 26173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:20.076797 26173 hybrid_clock.cc:648] HybridClock initialized: now 1790550560076760 us; error 51 us; skew 500 ppm
I20260927 23:09:20.077149 26173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:20 dist-test-slave-lmfp krb5kdc[26160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550560, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:20.080163 26173 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:09:20.081466 26173 webserver.cc:460] Webserver started at http://127.3.240.254:36599/ using document root <none> and password file <none>
I20260927 23:09:20.082109 26173 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:20.082247 26173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:20.082532 26173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:20.085325 26173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "bd5ac6d170a845c88434e3df2f72ae25"
format_stamp: "Formatted at 2026-09-27 23:09:20 on dist-test-slave-lmfp"
server_key: "5d5e6a4844ebea5c15b96a50d15b58ab"
server_key_iv: "0e79bb6ca43b97025ba458dee5ecc0ed"
server_key_version: "encryptionkey@0"
I20260927 23:09:20.086041 26173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "bd5ac6d170a845c88434e3df2f72ae25"
format_stamp: "Formatted at 2026-09-27 23:09:20 on dist-test-slave-lmfp"
server_key: "5d5e6a4844ebea5c15b96a50d15b58ab"
server_key_iv: "0e79bb6ca43b97025ba458dee5ecc0ed"
server_key_version: "encryptionkey@0"
I20260927 23:09:20.090924 26173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 23:09:20.094126 26188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:20.095335 26173 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260927 23:09:20.095492 26173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "bd5ac6d170a845c88434e3df2f72ae25"
format_stamp: "Formatted at 2026-09-27 23:09:20 on dist-test-slave-lmfp"
server_key: "5d5e6a4844ebea5c15b96a50d15b58ab"
server_key_iv: "0e79bb6ca43b97025ba458dee5ecc0ed"
server_key_version: "encryptionkey@0"
I20260927 23:09:20.095680 26173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:20.108180 26173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:20.111723 26173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:20.111972 26173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:20.129472 26173 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:45675
I20260927 23:09:20.129494 26239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:45675 every 8 connection(s)
I20260927 23:09:20.130805 26173 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:20.131740 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26173
I20260927 23:09:20.131901 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:20.132441 4035 external_mini_cluster.cc:1428] Setting key 777440626ec1c0763f93407afb717281
I20260927 23:09:20.135257 26240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:20.145284 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:20.134172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49049 (local address 127.3.240.254:45675)
0927 23:09:20.134904 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:20.134916 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:20.134955 (+ 39us) server_negotiation.cc:407] Connection header received
0927 23:09:20.136558 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:20.136578 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:20.136929 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:20.137235 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:20.137819 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:20.139225 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:20.141071 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:20.141712 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:20.141840 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:20.143144 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:20.143157 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:20.143170 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:20.143679 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:20.143758 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:20.143783 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:20.143871 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:20.144097 (+ 226us) server_negotiation.cc:299] Negotiation successful
0927 23:09:20.144378 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":339,"threads_started":1}
I20260927 23:09:20.147162 26240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25: Bootstrap starting.
I20260927 23:09:20.148653 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:20.150157 26240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:20.151266 26240 log.cc:824] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:20.154119 26240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25: No bootstrap required, opened a new log
I20260927 23:09:20.158751 26243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:20.152610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55987 (local address 127.3.240.254:45675)
0927 23:09:20.152765 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:20.152770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:20.153318 (+ 548us) server_negotiation.cc:407] Connection header received
0927 23:09:20.153457 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:20.153462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:20.153536 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:20.153632 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:20.154048 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:20.155057 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:20.156188 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:20.156911 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:20.157053 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:20.158022 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:20.158028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:20.158033 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:20.158053 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:20.158094 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:20.158113 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:20.158195 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:20.158326 (+ 131us) server_negotiation.cc:299] Negotiation successful
0927 23:09:20.158403 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260927 23:09:20.162302 26240 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5ac6d170a845c88434e3df2f72ae25" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45675 } }
I20260927 23:09:20.162671 26240 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:20.162782 26240 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd5ac6d170a845c88434e3df2f72ae25, State: Initialized, Role: FOLLOWER
I20260927 23:09:20.163362 26240 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5ac6d170a845c88434e3df2f72ae25" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45675 } }
I20260927 23:09:20.163575 26240 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:20.163715 26240 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:20.163880 26240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:20.165864 26240 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5ac6d170a845c88434e3df2f72ae25" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45675 } }
I20260927 23:09:20.166386 26240 leader_election.cc:302] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd5ac6d170a845c88434e3df2f72ae25; no voters:
I20260927 23:09:20.166915 26240 leader_election.cc:288] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:20.167044 26250 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:20.168344 26250 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [term 1 LEADER]: Becoming Leader. State: Replica: bd5ac6d170a845c88434e3df2f72ae25, State: Running, Role: LEADER
I20260927 23:09:20.168848 26250 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5ac6d170a845c88434e3df2f72ae25" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45675 } }
I20260927 23:09:20.169066 26240 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:20.172201 26252 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd5ac6d170a845c88434e3df2f72ae25. Latest consensus state: current_term: 1 leader_uuid: "bd5ac6d170a845c88434e3df2f72ae25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5ac6d170a845c88434e3df2f72ae25" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45675 } } }
I20260927 23:09:20.172411 26251 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd5ac6d170a845c88434e3df2f72ae25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd5ac6d170a845c88434e3df2f72ae25" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45675 } } }
I20260927 23:09:20.172632 26252 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:20.172688 26251 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25 [sys.catalog]: This master's current role is: LEADER
W20260927 23:09:20.180395 26266 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:09:20.180560 26266 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:09:20.180709 26259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:20.182313 26259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:20.190177 26259 catalog_manager.cc:1347] Generated new cluster ID: e95ff4e5f7e447f9a97db2bea9516884
I20260927 23:09:20.190281 26259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:20.212348 26259 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:20.214175 26259 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:20.236445 26259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd5ac6d170a845c88434e3df2f72ae25: Generated new TSK 0
I20260927 23:09:20.237466 26259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:20.248173 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26173
W20260927 23:09:20.249243 26235 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49049
W20260927 23:09:20.249910 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26173: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:09:20.250061 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26173
2026-09-27T23:09:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1809 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:20 dist-test-slave-lmfp krb5kdc[26271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:20 dist-test-slave-lmfp krb5kdc[26271](info): set up 2 sockets
Sep 27 23:09:20 dist-test-slave-lmfp krb5kdc[26271](info): commencing operation
krb5kdc: starting...
W20260927 23:09:21.323282 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:21 dist-test-slave-lmfp krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:09:21.570425 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39225
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:21.893514 26284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:21.893790 26284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:21.893957 26284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:21.900668 26284 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:21.900801 26284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:21.900906 26284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:21.901012 26284 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:21.901095 26284 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:21.910298 26284 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39225
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:21.911468 26284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:21.912822 26284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:21.924556 26292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:21.924680 26289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:21.924717 26290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:21.925534 26284 server_base.cc:1034] running on GCE node
I20260927 23:09:21.926288 26284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:21.927666 26284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:21.928877 26284 hybrid_clock.cc:648] HybridClock initialized: now 1790550561928841 us; error 36 us; skew 500 ppm
I20260927 23:09:21.929214 26284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:21 dist-test-slave-lmfp krb5kdc[26271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550561, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:21.937604 26284 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:09:21.939214 26284 webserver.cc:460] Webserver started at http://127.3.240.254:43275/ using document root <none> and password file <none>
I20260927 23:09:21.939827 26284 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:21.939930 26284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:21.940198 26284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:21.943195 26284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05"
format_stamp: "Formatted at 2026-09-27 23:09:21 on dist-test-slave-lmfp"
server_key: "354b91c5a84a0f6d18095f9f53d8cf52"
server_key_iv: "fe12b4cdca13c794f99d4183b86a23e8"
server_key_version: "encryptionkey@0"
I20260927 23:09:21.943933 26284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05"
format_stamp: "Formatted at 2026-09-27 23:09:21 on dist-test-slave-lmfp"
server_key: "354b91c5a84a0f6d18095f9f53d8cf52"
server_key_iv: "fe12b4cdca13c794f99d4183b86a23e8"
server_key_version: "encryptionkey@0"
I20260927 23:09:21.949070 26284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260927 23:09:21.952405 26299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:21.953612 26284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260927 23:09:21.953799 26284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05"
format_stamp: "Formatted at 2026-09-27 23:09:21 on dist-test-slave-lmfp"
server_key: "354b91c5a84a0f6d18095f9f53d8cf52"
server_key_iv: "fe12b4cdca13c794f99d4183b86a23e8"
server_key_version: "encryptionkey@0"
I20260927 23:09:21.954005 26284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:21.959815 26284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:21.963640 26284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:21.963932 26284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:21.981175 26284 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39225
I20260927 23:09:21.981245 26350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39225 every 8 connection(s)
I20260927 23:09:21.982410 26284 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:21.985743 26351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:21.991976 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26284
I20260927 23:09:21.992147 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:21.992648 4035 external_mini_cluster.cc:1428] Setting key 1f61bbef82602547322375b579f2e578
I20260927 23:09:21.995741 26351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05: Bootstrap starting.
I20260927 23:09:21.999354 26351 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:21.999964 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:21.994266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52695 (local address 127.3.240.254:39225)
0927 23:09:21.995039 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:21.995063 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:21.995101 (+ 38us) server_negotiation.cc:407] Connection header received
0927 23:09:21.996708 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:21.996729 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:21.997029 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:21.997354 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:21.998408 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:21.998414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:21.998425 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:21.998439 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:21.998490 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:21.998507 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:21.998668 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:21.998807 (+ 139us) server_negotiation.cc:299] Negotiation successful
0927 23:09:21.998983 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":339,"threads_started":1}
I20260927 23:09:22.000455 26351 log.cc:824] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:22.003379 26351 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05: No bootstrap required, opened a new log
I20260927 23:09:22.003701 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:22.010231 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:22.007960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37855 (local address 127.3.240.254:39225)
0927 23:09:22.008160 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:22.008167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:22.008481 (+ 314us) server_negotiation.cc:407] Connection header received
0927 23:09:22.008620 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:22.008624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:22.008663 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:22.008740 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:22.009618 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:22.009622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:22.009625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0927 23:09:22.009638 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:22.009670 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0927 23:09:22.009683 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:22.009778 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:22.009912 (+ 134us) server_negotiation.cc:299] Negotiation successful
0927 23:09:22.009949 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:09:22.011703 26351 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39225 } }
I20260927 23:09:22.012005 26351 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260927 23:09:22.012111 26351 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8cfa7f8bb7746a8a1e93eee372ecd05, State: Initialized, Role: FOLLOWER
I20260927 23:09:22.012596 26351 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39225 } }
I20260927 23:09:22.012840 26351 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260927 23:09:22.012996 26351 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260927 23:09:22.013202 26351 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:22.015271 26351 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39225 } }
I20260927 23:09:22.015697 26351 leader_election.cc:302] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8cfa7f8bb7746a8a1e93eee372ecd05; no voters:
I20260927 23:09:22.016353 26351 leader_election.cc:288] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260927 23:09:22.016449 26361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:22.017145 26361 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [term 1 LEADER]: Becoming Leader. State: Replica: f8cfa7f8bb7746a8a1e93eee372ecd05, State: Running, Role: LEADER
I20260927 23:09:22.017616 26361 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39225 } }
I20260927 23:09:22.018147 26351 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:22.020288 26362 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39225 } } }
I20260927 23:09:22.020709 26362 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:22.022086 26369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:22.022060 26363 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8cfa7f8bb7746a8a1e93eee372ecd05. Latest consensus state: current_term: 1 leader_uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8cfa7f8bb7746a8a1e93eee372ecd05" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39225 } } }
I20260927 23:09:22.022462 26363 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:22.027233 26369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:22.035657 26369 catalog_manager.cc:1347] Generated new cluster ID: e04db354dfdd414dbed4a87bcff3df3b
I20260927 23:09:22.035769 26369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:22.055972 26369 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:22.056403 26369 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:22.062847 26369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8cfa7f8bb7746a8a1e93eee372ecd05: Generated new TSK 0
I20260927 23:09:22.063673 26369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:22.070709 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26284
W20260927 23:09:22.071718 26346 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52695
W20260927 23:09:22.072373 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26284: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260927 23:09:22.072531 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26284
2026-09-27T23:09:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1822 ms)
[----------] 18 tests from AuthTokenIssuingTest (33256 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:22 dist-test-slave-lmfp krb5kdc[26382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:22 dist-test-slave-lmfp krb5kdc[26382](info): set up 2 sockets
Sep 27 23:09:22 dist-test-slave-lmfp krb5kdc[26382](info): commencing operation
krb5kdc: starting...
W20260927 23:09:23.151566 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:23 dist-test-slave-lmfp krb5kdc[26382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:09:23.400856 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44121
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44121,127.3.240.253:33593,127.3.240.252:35147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:23.723984 26395 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 23:09:23.724298 26395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:23.724368 26395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:23.724457 26395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:23.731781 26395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:23.731873 26395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:23.731930 26395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:23.731981 26395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:23.732036 26395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:23.741859 26395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44121,127.3.240.253:33593,127.3.240.252:35147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:23.743127 26395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:23.744379 26395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:23.756116 26400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:23.756076 26403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:23.756124 26401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:23.757014 26395 server_base.cc:1034] running on GCE node
I20260927 23:09:23.757690 26395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:23.758929 26395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:23.760186 26395 hybrid_clock.cc:648] HybridClock initialized: now 1790550563760132 us; error 26 us; skew 500 ppm
I20260927 23:09:23.760510 26395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:23 dist-test-slave-lmfp krb5kdc[26382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:23.763734 26395 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:09:23.765039 26395 webserver.cc:460] Webserver started at http://127.3.240.254:33543/ using document root <none> and password file <none>
I20260927 23:09:23.765661 26395 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:23.765743 26395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:23.765997 26395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:23.769032 26395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "0ba63f211cde43468557279127dc2e15"
format_stamp: "Formatted at 2026-09-27 23:09:23 on dist-test-slave-lmfp"
server_key: "1819a3c69ea24decb9209513e671b30c"
server_key_iv: "8f00f2adee25b1b2984df0e9fd5f1dfc"
server_key_version: "encryptionkey@0"
I20260927 23:09:23.770018 26395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "0ba63f211cde43468557279127dc2e15"
format_stamp: "Formatted at 2026-09-27 23:09:23 on dist-test-slave-lmfp"
server_key: "1819a3c69ea24decb9209513e671b30c"
server_key_iv: "8f00f2adee25b1b2984df0e9fd5f1dfc"
server_key_version: "encryptionkey@0"
I20260927 23:09:23.775096 26395 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260927 23:09:23.778460 26410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:23.779875 26395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:09:23.780051 26395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "0ba63f211cde43468557279127dc2e15"
format_stamp: "Formatted at 2026-09-27 23:09:23 on dist-test-slave-lmfp"
server_key: "1819a3c69ea24decb9209513e671b30c"
server_key_iv: "8f00f2adee25b1b2984df0e9fd5f1dfc"
server_key_version: "encryptionkey@0"
I20260927 23:09:23.780269 26395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:23.805322 26395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:23.809868 26395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:23.810163 26395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:23.828474 26395 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:44121
I20260927 23:09:23.828493 26461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:44121 every 8 connection(s)
I20260927 23:09:23.829775 26395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:23.833389 26462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:23.835103 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26395
I20260927 23:09:23.835312 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:23.836108 4035 external_mini_cluster.cc:1428] Setting key 323389ecb48867c6930abf39cc5b9926
I20260927 23:09:23.842725 26462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:23 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.253@KRBTEST.COM" created.
I20260927 23:09:23.865653 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:23.846635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44121 (local address 127.0.0.1:39684)
0927 23:09:23.847548 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:23.847598 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:23.849334 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:23.850439 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:23.850446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:23.851180 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:23.851486 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:23.851502 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:23.852959 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:23.852968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:23.854000 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:23.854011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:23.854892 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:23.854899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:23.855105 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:23.855747 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:23.855767 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:23.858106 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:23.861588 (+ 3482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:23.861601 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:23.861616 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:23.862177 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:23.862612 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:23.862615 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:23.862617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:23.862771 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:23.863687 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:23.863692 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:23.863874 (+ 182us) client_negotiation.cc:769] Sending connection context
0927 23:09:23.864068 (+ 194us) client_negotiation.cc:240] Negotiation successful
0927 23:09:23.864667 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"mutex_wait_us":125,"thread_start_us":446,"threads_started":1}
I20260927 23:09:23.865739 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:23.846635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.3.240.254:44121)
0927 23:09:23.849895 (+ 3260us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:23.849900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:23.849924 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:09:23.850031 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:23.850036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:23.850220 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:23.850403 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:23.851695 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:23.852807 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:23.854145 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:23.854677 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:23.854862 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:23.858394 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:23.858406 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:23.858424 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:23.858469 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:23.861438 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:23.862332 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:23.862342 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:23.862347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:23.862499 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:23.862876 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:23.862879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:23.862881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:23.863556 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:23.863725 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:23.864266 (+ 541us) server_negotiation.cc:299] Negotiation successful
0927 23:09:23.864498 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":3110,"spinlock_wait_cycles":13696,"thread_start_us":2933,"threads_started":1}
I20260927 23:09:23.868577 26462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:09:23.869521 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:23.869168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:50920)
0927 23:09:23.869322 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:23.869428 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 23:09:23.869760 26412 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
I20260927 23:09:23.871349 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:23.870973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:50936)
0927 23:09:23.871181 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:23.871247 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260927 23:09:23.871529 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33593: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
I20260927 23:09:23.893726 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:33593
--webserver_interface=127.3.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44121,127.3.240.253:33593,127.3.240.252:35147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:09:23.920848 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } attempt: 1
I20260927 23:09:23.922170 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:23.921594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:50950)
0927 23:09:23.921839 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:23.921985 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 23:09:23.923179 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:23.922794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:50954)
0927 23:09:23.922934 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:23.923033 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260927 23:09:23.923353 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33593: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
I20260927 23:09:23.991600 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } attempt: 2
I20260927 23:09:23.992856 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:23.992280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:50960)
0927 23:09:23.992549 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:23.992674 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 23:09:23.993963 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:23.993544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:50976)
0927 23:09:23.993701 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:23.993805 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 23:09:23.994177 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33593: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
I20260927 23:09:24.085435 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } attempt: 3
I20260927 23:09:24.086719 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.086129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:50988)
0927 23:09:24.086385 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.086522 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260927 23:09:24.087821 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.087421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:51002)
0927 23:09:24.087582 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.087664 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 23:09:24.088003 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33593: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
W20260927 23:09:24.221505 26469 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 23:09:24.221818 26469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:24.221892 26469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:24.221972 26469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:24.228952 26469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:24.229038 26469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:24.229101 26469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:24.229146 26469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:24.229208 26469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:24.231262 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } attempt: 4
I20260927 23:09:24.232504 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.231902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:51012)
0927 23:09:24.232236 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.232341 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 23:09:24.233528 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.233156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:51016)
0927 23:09:24.233299 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.233390 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260927 23:09:24.233711 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:33593: Network error: Client connection negotiation failed: client connection to 127.3.240.253:33593: connect: Connection refused (error 111)
I20260927 23:09:24.239495 26469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44121,127.3.240.253:33593,127.3.240.252:35147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:24.240653 26469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:24.241863 26469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:24.253010 26477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:24.253032 26474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:24.253103 26475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:24.256428 26469 server_base.cc:1034] running on GCE node
I20260927 23:09:24.257118 26469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:24.258378 26469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:24.259613 26469 hybrid_clock.cc:648] HybridClock initialized: now 1790550564259561 us; error 30 us; skew 500 ppm
I20260927 23:09:24.259929 26469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:24.262948 26469 init.cc:377] Logged in from keytab as kudu/127.3.240.253@KRBTEST.COM (short username kudu)
I20260927 23:09:24.264482 26469 webserver.cc:460] Webserver started at http://127.3.240.253:43095/ using document root <none> and password file <none>
I20260927 23:09:24.265174 26469 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:24.265300 26469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:24.265611 26469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:24.268484 26469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/data/instance:
uuid: "254475b64f5b41568500d06a8ac57241"
format_stamp: "Formatted at 2026-09-27 23:09:24 on dist-test-slave-lmfp"
server_key: "52658c696fbd4d9af75afdd28c1b68c2"
server_key_iv: "d52a4b157458d25ea2968c1ab7d2cba6"
server_key_version: "encryptionkey@0"
I20260927 23:09:24.269240 26469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/wal/instance:
uuid: "254475b64f5b41568500d06a8ac57241"
format_stamp: "Formatted at 2026-09-27 23:09:24 on dist-test-slave-lmfp"
server_key: "52658c696fbd4d9af75afdd28c1b68c2"
server_key_iv: "d52a4b157458d25ea2968c1ab7d2cba6"
server_key_version: "encryptionkey@0"
I20260927 23:09:24.274271 26469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260927 23:09:24.278079 26484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:24.279469 26469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:09:24.279642 26469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/wal
uuid: "254475b64f5b41568500d06a8ac57241"
format_stamp: "Formatted at 2026-09-27 23:09:24 on dist-test-slave-lmfp"
server_key: "52658c696fbd4d9af75afdd28c1b68c2"
server_key_iv: "d52a4b157458d25ea2968c1ab7d2cba6"
server_key_version: "encryptionkey@0"
I20260927 23:09:24.279852 26469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:24.308491 26469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:24.312690 26469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:24.312965 26469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:24.331162 26469 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:33593
I20260927 23:09:24.331193 26535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:33593 every 8 connection(s)
I20260927 23:09:24.332453 26469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
I20260927 23:09:24.335914 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:24.336715 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26469
I20260927 23:09:24.336920 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-1/wal/instance
I20260927 23:09:24.337334 4035 external_mini_cluster.cc:1428] Setting key 784fa643459767b0dd70d7f8a63142e8
I20260927 23:09:24.342180 26536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.252@KRBTEST.COM" created.
I20260927 23:09:24.362958 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.344754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.3.240.254:44121)
0927 23:09:24.344991 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.344999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.346125 (+ 1126us) server_negotiation.cc:407] Connection header received
0927 23:09:24.348070 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.348078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.348169 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.348297 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.349720 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.350904 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.352409 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.353149 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.353289 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.357307 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.357312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.357316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.357348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.360064 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.360880 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.360884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.360886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.360994 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.361423 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.361427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.361429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.361707 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.361839 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.362445 (+ 606us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.362701 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260927 23:09:24.363933 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.345162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44121 (local address 127.0.0.1:39696)
0927 23:09:24.345943 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.345992 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:24.347799 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:24.348484 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:24.348492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:24.348990 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:24.349538 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.349556 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.351104 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.351108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.352262 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.352274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.353343 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.353348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.353553 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.354245 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:24.354268 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:24.357075 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:24.360214 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.360224 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.360242 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.360759 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.361090 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.361094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.361096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.361311 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.361825 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:24.361830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:24.362124 (+ 294us) client_negotiation.cc:769] Sending connection context
0927 23:09:24.362335 (+ 211us) client_negotiation.cc:240] Negotiation successful
0927 23:09:24.362805 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":340,"threads_started":1}
I20260927 23:09:24.365737 26536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:09:24.380527 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.366621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:51024)
0927 23:09:24.366775 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.366794 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:24.366931 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:24.367633 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:24.367637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:24.368083 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:24.368212 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.368220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.369542 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.369547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.370366 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.370376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.371195 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.371200 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.371375 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.371936 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:24.371952 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:24.373837 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:24.378188 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.378192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.378194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.378544 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.379037 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.379046 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.379049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.379194 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.379896 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:24.379899 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:24.380005 (+ 106us) client_negotiation.cc:769] Sending connection context
0927 23:09:24.380105 (+ 100us) client_negotiation.cc:240] Negotiation successful
0927 23:09:24.380273 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260927 23:09:24.381093 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.366678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.3.240.253:33593)
0927 23:09:24.367267 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.367272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.367293 (+ 21us) server_negotiation.cc:407] Connection header received
0927 23:09:24.367331 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.367336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.367494 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.367652 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.368383 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.369408 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.370495 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.371057 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.371193 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.374055 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.374069 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.374088 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.374127 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.378057 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.378693 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.378704 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.378710 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.378867 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.379306 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.379313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.379319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.379771 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.379966 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.380280 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.380580 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":306,"threads_started":1}
I20260927 23:09:24.383628 26536 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:09:24.384910 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.384335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38114)
0927 23:09:24.384526 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.384723 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260927 23:09:24.385200 26485 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
I20260927 23:09:24.386675 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.386363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38120)
0927 23:09:24.386508 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.386579 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260927 23:09:24.386965 26536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
I20260927 23:09:24.396045 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:35147
--webserver_interface=127.3.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:44833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44121,127.3.240.253:33593,127.3.240.252:35147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:09:24.436313 26536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 1
I20260927 23:09:24.437480 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.437039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38136)
0927 23:09:24.437237 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.437338 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 23:09:24.438540 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.438217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38152)
0927 23:09:24.438375 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.438444 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260927 23:09:24.438825 26536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
I20260927 23:09:24.507056 26536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 2
I20260927 23:09:24.508214 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.507787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38166)
0927 23:09:24.507965 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.508062 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 23:09:24.509224 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.508842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38172)
0927 23:09:24.508994 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.509055 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 23:09:24.509510 26536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
I20260927 23:09:24.532934 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } attempt: 5
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:09:24.546037 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.533626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:51028)
0927 23:09:24.533812 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.533836 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:24.533969 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:24.534354 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:24.534360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:24.534761 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:24.534947 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.534956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.535955 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.535961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.536813 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.536824 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.537593 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.537607 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.537832 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.538526 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:24.538546 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:24.540831 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:24.543734 (+ 2903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.543739 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.543744 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.544097 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.544447 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.544452 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.544456 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.544588 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.545105 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:24.545110 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:24.545266 (+ 156us) client_negotiation.cc:769] Sending connection context
0927 23:09:24.545408 (+ 142us) client_negotiation.cc:240] Negotiation successful
0927 23:09:24.545690 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 23:09:24.546028 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.533740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51028 (local address 127.3.240.253:33593)
0927 23:09:24.533940 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.533945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.533967 (+ 22us) server_negotiation.cc:407] Connection header received
0927 23:09:24.534111 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.534116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.534202 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.534311 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.535134 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.535822 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.536973 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.537443 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.537580 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.541017 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.541022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.541025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.541054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.543602 (+ 2548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.544210 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.544217 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.544221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.544340 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.544697 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.544704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.544707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.544962 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.545134 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.545500 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.545712 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260927 23:09:24.547341 26462 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:09:24.548398 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.547983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38174)
0927 23:09:24.548177 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.548252 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260927 23:09:24.549324 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.548999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38182)
0927 23:09:24.549168 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.549230 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260927 23:09:24.549527 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
I20260927 23:09:24.600759 26536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 3
I20260927 23:09:24.600759 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 1
I20260927 23:09:24.601882 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.601400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38192)
0927 23:09:24.601631 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.601719 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260927 23:09:24.601995 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.601415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38194)
0927 23:09:24.601698 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.601808 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260927 23:09:24.603070 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.602657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38214)
0927 23:09:24.602827 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.602914 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260927 23:09:24.603184 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.602632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38218)
0927 23:09:24.602911 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.603034 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
W20260927 23:09:24.603397 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
W20260927 23:09:24.603533 26536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
I20260927 23:09:24.671653 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 2
I20260927 23:09:24.672819 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.672301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38230)
0927 23:09:24.672549 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.672666 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260927 23:09:24.673792 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.673458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38238)
0927 23:09:24.673608 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.673682 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260927 23:09:24.674026 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
W20260927 23:09:24.718664 26543 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 23:09:24.718948 26543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:24.719079 26543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:24.719166 26543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:24.725821 26543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:24.725893 26543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:24.725943 26543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:24.725991 26543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:24.726045 26543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:24.734990 26543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44121,127.3.240.253:33593,127.3.240.252:35147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:35147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:24.736173 26543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:24.737381 26543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 23:09:24.746762 26536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 4
I20260927 23:09:24.748116 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.747459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38252)
0927 23:09:24.747798 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.747919 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20260927 23:09:24.748284 26548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:24.748828 26551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:24.748996 26549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:24.749411 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.748959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38264)
0927 23:09:24.749191 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.749279 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260927 23:09:24.749397 26543 server_base.cc:1034] running on GCE node
I20260927 23:09:24.750154 26543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260927 23:09:24.751169 26536 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
I20260927 23:09:24.751502 26543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:24.752730 26543 hybrid_clock.cc:648] HybridClock initialized: now 1790550564752671 us; error 37 us; skew 500 ppm
I20260927 23:09:24.753028 26543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:24.756021 26543 init.cc:377] Logged in from keytab as kudu/127.3.240.252@KRBTEST.COM (short username kudu)
I20260927 23:09:24.757315 26543 webserver.cc:460] Webserver started at http://127.3.240.252:40025/ using document root <none> and password file <none>
I20260927 23:09:24.757894 26543 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:24.757975 26543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:24.758229 26543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:24.760885 26543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/data/instance:
uuid: "47ddbc00cdf74791927be05b72dc9a6a"
format_stamp: "Formatted at 2026-09-27 23:09:24 on dist-test-slave-lmfp"
server_key: "9c3648d3b6066137d82a4d963996c2ac"
server_key_iv: "1d5567e6d215c4e8cfe1fb075f0b7722"
server_key_version: "encryptionkey@0"
I20260927 23:09:24.761548 26543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/wal/instance:
uuid: "47ddbc00cdf74791927be05b72dc9a6a"
format_stamp: "Formatted at 2026-09-27 23:09:24 on dist-test-slave-lmfp"
server_key: "9c3648d3b6066137d82a4d963996c2ac"
server_key_iv: "1d5567e6d215c4e8cfe1fb075f0b7722"
server_key_version: "encryptionkey@0"
I20260927 23:09:24.766124 26543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260927 23:09:24.769130 26558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:24.770252 26543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260927 23:09:24.770370 26543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/wal
uuid: "47ddbc00cdf74791927be05b72dc9a6a"
format_stamp: "Formatted at 2026-09-27 23:09:24 on dist-test-slave-lmfp"
server_key: "9c3648d3b6066137d82a4d963996c2ac"
server_key_iv: "1d5567e6d215c4e8cfe1fb075f0b7722"
server_key_version: "encryptionkey@0"
I20260927 23:09:24.770516 26543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:24.780226 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 3
I20260927 23:09:24.781232 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.780776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38276)
0927 23:09:24.780994 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.781066 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260927 23:09:24.782222 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.781862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38292)
0927 23:09:24.782058 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.782134 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260927 23:09:24.782449 26462 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:35147: Network error: Client connection negotiation failed: client connection to 127.3.240.252:35147: connect: Connection refused (error 111)
I20260927 23:09:24.807160 26543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:24.811611 26543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:24.811869 26543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:24.829138 26543 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:35147
I20260927 23:09:24.829159 26609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:35147 every 8 connection(s)
I20260927 23:09:24.830385 26543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
I20260927 23:09:24.833912 26610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:24.839545 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26543
I20260927 23:09:24.839707 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790550224434615-4035-0/minicluster-data/master-2/wal/instance
I20260927 23:09:24.840126 4035 external_mini_cluster.cc:1428] Setting key b61c62f99c2c4b1df20067bc13bce886
I20260927 23:09:24.839977 26610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260927 23:09:24.859725 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.842548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.3.240.254:44121)
0927 23:09:24.842759 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.842765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.843701 (+ 936us) server_negotiation.cc:407] Connection header received
0927 23:09:24.844653 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.844658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.844736 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.844858 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.846454 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.847500 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.849373 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.849897 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.850005 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.853670 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.853674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.853677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.853704 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.856402 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.857292 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.857295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.857297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.857414 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.857859 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.857864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.857867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.858210 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.858350 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.859154 (+ 804us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.859362 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260927 23:09:24.860670 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.841810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.3.240.252:35147)
0927 23:09:24.842527 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.842542 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.842584 (+ 42us) server_negotiation.cc:407] Connection header received
0927 23:09:24.844410 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.844430 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.844719 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.844909 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.846613 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.847977 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.849551 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.850302 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.850385 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.854473 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.854479 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.854488 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.854514 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.857556 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.858246 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.858251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.858253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.858402 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.858737 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.858742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.858744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.859171 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.859437 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.860132 (+ 695us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.860302 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":524,"thread_start_us":336,"threads_started":1}
I20260927 23:09:24.860690 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.842651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44121 (local address 127.0.0.1:39698)
0927 23:09:24.843543 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.843585 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:24.844421 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:24.844932 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:24.844938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:24.845808 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:24.846290 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.846312 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.847665 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.847669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.849233 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.849247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.850073 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.850077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.850216 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.850892 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:24.850920 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:24.853489 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:24.856521 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.856530 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.856540 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.857175 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.857515 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.857518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.857521 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.857741 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.858341 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:24.858351 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:24.858701 (+ 350us) client_negotiation.cc:769] Sending connection context
0927 23:09:24.859037 (+ 336us) client_negotiation.cc:240] Negotiation successful
0927 23:09:24.859468 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":17,"thread_start_us":459,"threads_started":1}
I20260927 23:09:24.862118 26610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:09:24.875381 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.862897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:33593 (local address 127.0.0.1:51036)
0927 23:09:24.863165 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.863178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:24.863288 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:24.863671 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:24.863675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:24.864124 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:24.864306 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.864319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.865186 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.865190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.866025 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.866038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.866740 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.866747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.866960 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.867566 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:24.867581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:24.869643 (+ 2062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:24.872827 (+ 3184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.872833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.872837 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.873345 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.873726 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.873730 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.873734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.873885 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.874492 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:24.874496 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:24.874640 (+ 144us) client_negotiation.cc:769] Sending connection context
0927 23:09:24.874762 (+ 122us) client_negotiation.cc:240] Negotiation successful
0927 23:09:24.875048 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260927 23:09:24.875454 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.862821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.3.240.253:33593)
0927 23:09:24.863092 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.863097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.863242 (+ 145us) server_negotiation.cc:407] Connection header received
0927 23:09:24.863442 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.863447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.863530 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.863646 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.864439 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.865060 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.866178 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.866605 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.866709 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.869828 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.869836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.869844 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.869888 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.872704 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.873478 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.873484 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.873487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.873615 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.873996 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.874001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.874004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.874373 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.874523 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.874857 (+ 334us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.875109 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 23:09:24.876436 26610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:09:24.876847 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.863703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.3.240.254:44121)
0927 23:09:24.863841 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.863845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.863875 (+ 30us) server_negotiation.cc:407] Connection header received
0927 23:09:24.864035 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.864038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.864081 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.864153 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.865292 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.865905 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.867274 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.867781 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.867892 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.871749 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.871754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.871759 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.871793 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.874284 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.874870 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.874876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.874879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.875080 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.875429 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.875435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.875438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.875739 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.876006 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.876247 (+ 241us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.876457 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:09:24.889621 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.877286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.3.240.252:35147)
0927 23:09:24.877703 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.877709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.877723 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:09:24.877766 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.877770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.877833 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.877899 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.878708 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.879451 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.880540 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.880925 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.881054 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.884314 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.884319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.884322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.884348 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.886788 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.887590 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.887598 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.887602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.887761 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.888164 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.888167 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.888169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.888456 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.888598 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.889027 (+ 429us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.889369 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":177,"server-negotiator.queue_time_us":320}
I20260927 23:09:24.889698 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.877067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38298)
0927 23:09:24.877213 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.877230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:24.877313 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:24.877965 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:24.877970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:24.878405 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:24.878598 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.878611 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.879577 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.879582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.880411 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.880421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.881059 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.881065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.881298 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.881840 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:24.881855 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:24.884141 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:24.886906 (+ 2765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.886913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.886918 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.887478 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.887871 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.887878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.887883 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.888046 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.888593 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:24.888598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:24.888778 (+ 180us) client_negotiation.cc:769] Sending connection context
0927 23:09:24.888935 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 23:09:24.889243 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260927 23:09:24.892930 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.878478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.3.240.253:33593)
0927 23:09:24.878625 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.878629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.878642 (+ 13us) server_negotiation.cc:407] Connection header received
0927 23:09:24.878676 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.878679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.878724 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.878823 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.879984 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.880670 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.881978 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.882381 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.882511 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.886763 (+ 4252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.886768 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.886772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.886809 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.890188 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.891067 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.891071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.891075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.891232 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.891602 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.891607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.891610 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.891850 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.892127 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.892379 (+ 252us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.892640 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260927 23:09:24.895130 26610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: Bootstrap starting.
I20260927 23:09:24.897372 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:24.898676 26610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:24.899876 26610 log.cc:824] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:24.899885 26437 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0ba63f211cde43468557279127dc2e15"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39704
I20260927 23:09:24.902426 26610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: No bootstrap required, opened a new log
I20260927 23:09:24.909221 26610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } }
I20260927 23:09:24.909528 26610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47ddbc00cdf74791927be05b72dc9a6a, State: Initialized, Role: FOLLOWER
I20260927 23:09:24.909936 26610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } }
I20260927 23:09:24.910921 26616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:24.911309 26616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:24.912055 26610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [sys.catalog]: configured and running, proceeding with master startup.
W20260927 23:09:24.920953 26627 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:09:24.921116 26627 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:09:24.918725 26543 master_runner.cc:186] Error getting master registration for 127.3.240.254:44121: OK, instance_id { permanent_uuid: "0ba63f211cde43468557279127dc2e15" instance_seqno: 1790550563823337 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 23:09:24.929697 26543 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 23:09:24.959681 26462 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 4
Sep 27 23:09:24 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550563, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260927 23:09:24.975984 26465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.960441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38300)
0927 23:09:24.960607 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:24.960627 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:24.960729 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:24.961066 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:24.961070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:24.961597 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:24.961800 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.961811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.963198 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.963203 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.964575 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:24.964592 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.965488 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.965494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:24.965682 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.966317 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:24.966341 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:24.969174 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:24.973248 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.973254 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.973258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.973783 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.974142 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:24.974148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:24.974151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:24.974310 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:24.975018 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:24.975024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:24.975202 (+ 178us) client_negotiation.cc:769] Sending connection context
0927 23:09:24.975359 (+ 157us) client_negotiation.cc:240] Negotiation successful
0927 23:09:24.975621 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260927 23:09:24.975996 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:24.960521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.3.240.252:35147)
0927 23:09:24.960705 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:24.960711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:24.960732 (+ 21us) server_negotiation.cc:407] Connection header received
0927 23:09:24.960833 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:24.960838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:24.960914 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:24.961023 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:24.961957 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.963059 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.964720 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:24.965349 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:24.965478 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:24.969380 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:24.969386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:24.969391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:24.969449 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:24.973114 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.973887 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.973893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.973898 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.974034 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:24.974418 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:24.974423 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:24.974426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:24.974869 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:24.975043 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:24.975463 (+ 420us) server_negotiation.cc:299] Negotiation successful
0927 23:09:24.975669 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260927 23:09:24.982010 26462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15: Bootstrap starting.
I20260927 23:09:24.986120 26462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:24.987448 26462 log.cc:824] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:24.990578 26462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15: No bootstrap required, opened a new log
I20260927 23:09:24.997947 26462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } }
I20260927 23:09:24.998348 26462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba63f211cde43468557279127dc2e15, State: Initialized, Role: FOLLOWER
I20260927 23:09:24.998859 26462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } }
I20260927 23:09:24.999884 26630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.000252 26630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:25.001129 26462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:25.001184 26437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:09:25.001390 26437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:25.003841 26437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } }
I20260927 23:09:25.005249 26437 leader_election.cc:288] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [CANDIDATE]: Term 1 election: Requested vote from peers 254475b64f5b41568500d06a8ac57241 (127.3.240.253:33593), 47ddbc00cdf74791927be05b72dc9a6a (127.3.240.252:35147)
I20260927 23:09:25.005864 26585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ba63f211cde43468557279127dc2e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "47ddbc00cdf74791927be05b72dc9a6a"
I20260927 23:09:25.006357 26585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:25.010000 26585 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ba63f211cde43468557279127dc2e15 in term 1.
I20260927 23:09:25.010573 26411 leader_election.cc:302] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba63f211cde43468557279127dc2e15, 47ddbc00cdf74791927be05b72dc9a6a; no voters:
I20260927 23:09:25.011044 26630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:25.019263 26630 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [term 1 LEADER]: Becoming Leader. State: Replica: 0ba63f211cde43468557279127dc2e15, State: Running, Role: LEADER
I20260927 23:09:25.020145 26630 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } }
I20260927 23:09:25.027984 26635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba63f211cde43468557279127dc2e15. Latest consensus state: current_term: 1 leader_uuid: "0ba63f211cde43468557279127dc2e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.028395 26635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:25.050377 26645 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:25.050377 26536 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } attempt: 5
W20260927 23:09:25.062883 26416 tablet.cc:2369] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260927 23:09:25.063822 26564 tablet.cc:2369] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:09:25.076035 26645 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 27 23:09:25 dist-test-slave-lmfp krb5kdc[26382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550564, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260927 23:09:25.079722 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:25.029540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.3.240.253:33593)
0927 23:09:25.029777 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.029786 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.050657 (+ 20871us) server_negotiation.cc:407] Connection header received
0927 23:09:25.051195 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.051204 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.051320 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.051406 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.058112 (+ 6706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.059623 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.061558 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.062254 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.062370 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:25.067712 (+ 5342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:25.067718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:25.067722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:25.067764 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:25.071723 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:25.074241 (+ 2518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:25.074249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:25.074252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:25.074419 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:25.074873 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:25.074879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:25.074882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:25.077472 (+ 2590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:25.077823 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:25.078871 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0927 23:09:25.079192 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260927 23:09:25.082754 26653 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 23:09:25.083053 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:25.036590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38312 (local address 127.3.240.252:35147)
0927 23:09:25.036755 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.036761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.054661 (+ 17900us) server_negotiation.cc:407] Connection header received
0927 23:09:25.054939 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.054947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.055069 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.055154 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.061619 (+ 6465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.062720 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.067718 (+ 4998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.068473 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.068590 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:25.071799 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:25.071805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:25.071810 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:25.071852 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:25.075485 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:25.076231 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:25.076236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:25.076241 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:25.076395 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:25.076935 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:25.076943 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:25.076949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:25.077362 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:25.077646 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:25.082372 (+ 4726us) server_negotiation.cc:299] Negotiation successful
0927 23:09:25.082642 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260927 23:09:25.088799 26466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:25.029710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.3.240.254:44121)
0927 23:09:25.029966 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.029974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.044182 (+ 14208us) server_negotiation.cc:407] Connection header received
0927 23:09:25.044389 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.044398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.044503 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.044622 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.059127 (+ 14505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.060567 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.071103 (+ 10536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.071815 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.071910 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:25.081119 (+ 9209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:25.081127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:25.081132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:25.081194 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:25.085275 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:25.086112 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:25.086117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:25.086120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:25.086283 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:25.087105 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:25.087111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:25.087115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:25.087453 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:25.087699 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:25.088229 (+ 530us) server_negotiation.cc:299] Negotiation successful
0927 23:09:25.088451 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260927 23:09:25.088948 26539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:25.051490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:35147 (local address 127.0.0.1:38314)
0927 23:09:25.051760 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:25.051779 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:25.051875 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:25.066507 (+ 14632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:25.066512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:25.067038 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:25.067259 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:25.067271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.070258 (+ 2987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.070264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:25.071793 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:25.071808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.073427 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.073433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:25.073669 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:25.074453 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:25.074474 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:25.077273 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:25.085924 (+ 8651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:25.085929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:25.085933 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:25.086444 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:25.087184 (+ 740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:25.087191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:25.087195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:25.087357 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:25.088007 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:25.088011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:25.088150 (+ 139us) client_negotiation.cc:769] Sending connection context
0927 23:09:25.088294 (+ 144us) client_negotiation.cc:240] Negotiation successful
0927 23:09:25.088556 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260927 23:09:25.088994 26650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:25.051239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.3.240.252:35147)
0927 23:09:25.065977 (+ 14738us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.066011 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.066034 (+ 23us) server_negotiation.cc:407] Connection header received
0927 23:09:25.066145 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.066151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.066274 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.067465 (+ 1191us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.067925 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.070015 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.072073 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:25.073199 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:25.077938 (+ 4739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:25.078340 (+ 402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:25.078373 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:25.078377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:25.078432 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:25.085768 (+ 7336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:25.086696 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:25.086701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:25.086705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:25.086857 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:25.087490 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:25.087498 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:25.087501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:25.087883 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:25.088027 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:25.088394 (+ 367us) server_negotiation.cc:299] Negotiation successful
0927 23:09:25.088585 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14556,"thread_start_us":14242,"threads_started":1}
I20260927 23:09:25.094586 26536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241: Bootstrap starting.
I20260927 23:09:25.099371 26536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:25.100622 26536 log.cc:824] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:25.105455 26585 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [term 1 FOLLOWER]: Refusing update from remote peer 0ba63f211cde43468557279127dc2e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:09:25.087203 26395 master_runner.cc:186] Error getting master registration for 127.3.240.253:33593: OK, instance_id { permanent_uuid: "254475b64f5b41568500d06a8ac57241" instance_seqno: 1790550564325961 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 23:09:25.106276 26395 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 23:09:25.106375 26635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:09:25.110077 26536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241: No bootstrap required, opened a new log
I20260927 23:09:25.122160 26536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } }
I20260927 23:09:25.122568 26536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254475b64f5b41568500d06a8ac57241, State: Initialized, Role: FOLLOWER
I20260927 23:09:25.123134 26536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } }
I20260927 23:09:25.130020 26658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.130385 26658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:25.132030 26536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:25.132874 26616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba63f211cde43468557279127dc2e15. Latest consensus state: current_term: 1 leader_uuid: "0ba63f211cde43468557279127dc2e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.133206 26616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:25.134806 26510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:25.136199 26654 mvcc.cc:205] Tried to move back new op lower bound from 7334095114660614144 to 7334095114328100864. Current Snapshot: MvccSnapshot[applied={T|T < 7334095114660614144}]
I20260927 23:09:25.139360 26510 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [term 1 FOLLOWER]: Refusing update from remote peer 0ba63f211cde43468557279127dc2e15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:09:25.139721 26656 mvcc.cc:205] Tried to move back new op lower bound from 7334095114660614144 to 7334095114328100864. Current Snapshot: MvccSnapshot[applied={T|T < 7334095114660614144}]
I20260927 23:09:25.140375 26635 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:09:25.141402 26511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ba63f211cde43468557279127dc2e15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "254475b64f5b41568500d06a8ac57241"
I20260927 23:09:25.146145 26616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ba63f211cde43468557279127dc2e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.146426 26616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:25.147516 26511 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ba63f211cde43468557279127dc2e15 in term 1.
I20260927 23:09:25.150770 26630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ba63f211cde43468557279127dc2e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.151151 26630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:25.153609 26645 catalog_manager.cc:1347] Generated new cluster ID: 896821747e6c434f8313ad48f51614d8
I20260927 23:09:25.153704 26645 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:25.184424 26645 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:25.186525 26645 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:25.203065 26658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ba63f211cde43468557279127dc2e15. Latest consensus state: current_term: 1 leader_uuid: "0ba63f211cde43468557279127dc2e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.203075 26667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ba63f211cde43468557279127dc2e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.203464 26658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:25.203466 26667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:25.204838 26635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ba63f211cde43468557279127dc2e15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ba63f211cde43468557279127dc2e15" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44121 } } peers { permanent_uuid: "254475b64f5b41568500d06a8ac57241" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 33593 } } peers { permanent_uuid: "47ddbc00cdf74791927be05b72dc9a6a" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 35147 } } }
I20260927 23:09:25.205133 26635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:25.213851 26676 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260927 23:09:25.220214 26645 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ba63f211cde43468557279127dc2e15: Generated new TSK 0
I20260927 23:09:25.221225 26645 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260927 23:09:25.286422 26466 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.274711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.3.240.254:44121)
0927 23:09:25.274959 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.274966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.284314 (+ 9348us) server_negotiation.cc:407] Connection header received
0927 23:09:25.284556 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.284562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.284652 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.284773 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.286168 (+ 1395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39724: BlockingRecv error: recv got EOF from 127.0.0.1:39724 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260927 23:09:25.289278 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.276022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38322 (local address 127.3.240.252:35147)
0927 23:09:25.276183 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.276190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.286326 (+ 10136us) server_negotiation.cc:407] Connection header received
0927 23:09:25.287941 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.287946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.288050 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.288137 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.289050 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38322: BlockingRecv error: recv got EOF from 127.0.0.1:38322 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260927 23:09:25.289415 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.275575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51044 (local address 127.3.240.253:33593)
0927 23:09:25.275837 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.275845 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.287615 (+ 11770us) server_negotiation.cc:407] Connection header received
0927 23:09:25.288178 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.288186 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.288280 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.288350 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.289200 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51044: BlockingRecv error: recv got EOF from 127.0.0.1:51044 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260927 23:09:25.295934 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.294084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.3.240.253:33593)
0927 23:09:25.294272 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.294278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.294567 (+ 289us) server_negotiation.cc:407] Connection header received
0927 23:09:25.294801 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.294805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.294898 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.295024 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.295694 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51060: BlockingRecv error: recv got EOF from 127.0.0.1:51060 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260927 23:09:25.312500 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.308652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51066 (local address 127.3.240.253:33593)
0927 23:09:25.308845 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.308853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.311257 (+ 2404us) server_negotiation.cc:407] Connection header received
0927 23:09:25.311450 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.311454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.311514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.311614 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.312297 (+ 683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51066: BlockingRecv error: recv got EOF from 127.0.0.1:51066 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260927 23:09:25.324803 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.318289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.3.240.253:33593)
0927 23:09:25.318480 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.318489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.323512 (+ 5023us) server_negotiation.cc:407] Connection header received
0927 23:09:25.323731 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.323734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.323809 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.323914 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.324570 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51070: BlockingRecv error: recv got EOF from 127.0.0.1:51070 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 23:09:25.355370 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.345794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.3.240.253:33593)
0927 23:09:25.346023 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.346032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.353813 (+ 7781us) server_negotiation.cc:407] Connection header received
0927 23:09:25.354034 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.354039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.354150 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.354241 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.355159 (+ 918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51082: BlockingRecv error: recv got EOF from 127.0.0.1:51082 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 23:09:25.369418 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.367233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.3.240.253:33593)
0927 23:09:25.367481 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.367490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.368054 (+ 564us) server_negotiation.cc:407] Connection header received
0927 23:09:25.368267 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.368272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.368347 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.368455 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.369213 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51090: BlockingRecv error: recv got EOF from 127.0.0.1:51090 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260927 23:09:25.390678 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.388537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.3.240.253:33593)
0927 23:09:25.388728 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.388737 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.389255 (+ 518us) server_negotiation.cc:407] Connection header received
0927 23:09:25.389463 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.389470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.389554 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.389669 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.390450 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51106: BlockingRecv error: recv got EOF from 127.0.0.1:51106 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260927 23:09:25.428107 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.425893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51118 (local address 127.3.240.253:33593)
0927 23:09:25.426153 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.426161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.426679 (+ 518us) server_negotiation.cc:407] Connection header received
0927 23:09:25.426891 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.426895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.426963 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.427129 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.427831 (+ 702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51118: BlockingRecv error: recv got EOF from 127.0.0.1:51118 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260927 23:09:25.500324 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.498250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.3.240.253:33593)
0927 23:09:25.498469 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.498475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.498809 (+ 334us) server_negotiation.cc:407] Connection header received
0927 23:09:25.499106 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.499110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.499179 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.499255 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.500111 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51120: BlockingRecv error: recv got EOF from 127.0.0.1:51120 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260927 23:09:25.633104 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.631263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.3.240.253:33593)
0927 23:09:25.631481 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.631488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.631756 (+ 268us) server_negotiation.cc:407] Connection header received
0927 23:09:25.631964 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.631969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.632044 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.632152 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.632845 (+ 693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51128: BlockingRecv error: recv got EOF from 127.0.0.1:51128 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260927 23:09:25.894532 26540 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:25.892660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51130 (local address 127.3.240.253:33593)
0927 23:09:25.892914 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:25.892921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:25.893090 (+ 169us) server_negotiation.cc:407] Connection header received
0927 23:09:25.893339 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:25.893344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:25.893428 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:25.893538 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:25.894348 (+ 810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51130: BlockingRecv error: recv got EOF from 127.0.0.1:51130 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 23:09:25.922889 26627 catalog_manager.cc:1259] Loaded cluster ID: 896821747e6c434f8313ad48f51614d8
I20260927 23:09:25.922981 26627 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: loading cluster ID for follower catalog manager: success
I20260927 23:09:25.925377 26627 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: acquiring CA information for follower catalog manager: success
I20260927 23:09:25.926970 26627 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47ddbc00cdf74791927be05b72dc9a6a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 23:09:26.217996 26676 catalog_manager.cc:1259] Loaded cluster ID: 896821747e6c434f8313ad48f51614d8
I20260927 23:09:26.218120 26676 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241: loading cluster ID for follower catalog manager: success
I20260927 23:09:26.220715 26676 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241: acquiring CA information for follower catalog manager: success
I20260927 23:09:26.222604 26676 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 254475b64f5b41568500d06a8ac57241: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 23:09:26.419866 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:26.413670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49844 (local address 127.3.240.253:33593)
0927 23:09:26.414498 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:26.414503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:26.414517 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:09:26.414592 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:26.414596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:26.414723 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:26.414933 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:26.415371 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.416212 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.417497 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.417936 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.418094 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:26.418601 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:26.419210 (+ 609us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:26.419366 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:26.419470 (+ 104us) server_negotiation.cc:299] Negotiation successful
0927 23:09:26.419569 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":532,"threads_started":1}
I20260927 23:09:26.437259 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:26.426039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55520 (local address 127.3.240.252:35147)
0927 23:09:26.431364 (+ 5325us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:26.431369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:26.431384 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:09:26.431471 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:26.431475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:26.431611 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:26.431786 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:26.432318 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.433081 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.435090 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.435457 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.435537 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:26.436163 (+ 626us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:26.436602 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:26.436715 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:26.436894 (+ 179us) server_negotiation.cc:299] Negotiation successful
0927 23:09:26.436953 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5215,"thread_start_us":5115,"threads_started":1}
I20260927 23:09:26.461905 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:26.443733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.3.240.253:33593)
0927 23:09:26.443958 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:26.443965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:26.454239 (+ 10274us) server_negotiation.cc:407] Connection header received
0927 23:09:26.455083 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:26.455091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:26.455190 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:26.455251 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:26.456024 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.457184 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.459528 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.460203 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.460292 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:26.461082 (+ 790us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:26.461222 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:26.461282 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:26.461491 (+ 209us) server_negotiation.cc:299] Negotiation successful
0927 23:09:26.461569 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260927 23:09:26.462320 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:26.443934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.3.240.252:35147)
0927 23:09:26.444121 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:26.444129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:26.454698 (+ 10569us) server_negotiation.cc:407] Connection header received
0927 23:09:26.455072 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:26.455078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:26.455191 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:26.455266 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:26.456208 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.457302 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.459869 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.460509 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.460638 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:26.461345 (+ 707us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:26.461490 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:26.461575 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:26.461858 (+ 283us) server_negotiation.cc:299] Negotiation successful
0927 23:09:26.461954 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260927 23:09:26.462733 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:26.443661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.3.240.254:44121)
0927 23:09:26.454097 (+ 10436us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:26.454103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:26.455117 (+ 1014us) server_negotiation.cc:407] Connection header received
0927 23:09:26.455239 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:26.455244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:26.455457 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:26.455628 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:26.456421 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.457385 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.459547 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:26.460173 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:26.460293 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:26.461141 (+ 848us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:26.461559 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:26.461660 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:26.462126 (+ 466us) server_negotiation.cc:299] Negotiation successful
0927 23:09:26.462227 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10314,"thread_start_us":10367,"threads_started":1}
I20260927 23:09:26.468863 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26395
I20260927 23:09:26.469784 26457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:26.577294 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26395
W20260927 23:09:26.597206 26560 connection.cc:535] server connection from 127.0.0.1:38300 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260927 23:09:26.597563 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26469
I20260927 23:09:26.598626 26531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:26.701428 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26469
I20260927 23:09:26.721493 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26543
I20260927 23:09:26.722699 26605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:26.827844 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26543
2026-09-27T23:09:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4763 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 27 23:09:26 dist-test-slave-lmfp krb5kdc[26762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 27 23:09:26 dist-test-slave-lmfp krb5kdc[26762](info): set up 2 sockets
Sep 27 23:09:26 dist-test-slave-lmfp krb5kdc[26762](info): commencing operation
krb5kdc: starting...
W20260927 23:09:27.907295 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/test-user.keytab.
Sep 27 23:09:28 dist-test-slave-lmfp krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-27T23:09:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-27T23:09:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260927 23:09:28.152037 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45377
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45377,127.3.240.253:41399,127.3.240.252:45507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260927 23:09:28.484108 26775 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 23:09:28.484421 26775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:28.484508 26775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:28.484594 26775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:28.491688 26775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:28.491792 26775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:28.491856 26775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:28.491914 26775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:28.491976 26775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:28.501508 26775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45377,127.3.240.253:41399,127.3.240.252:45507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:28.502698 26775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:28.503973 26775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:28.515277 26781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:28.515362 26780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:28.515393 26783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:28.516362 26775 server_base.cc:1034] running on GCE node
I20260927 23:09:28.517012 26775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:28.518288 26775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:28.519484 26775 hybrid_clock.cc:648] HybridClock initialized: now 1790550568519436 us; error 32 us; skew 500 ppm
I20260927 23:09:28.519739 26775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:28 dist-test-slave-lmfp krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550568, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:28.528537 26775 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260927 23:09:28.529899 26775 webserver.cc:460] Webserver started at http://127.3.240.254:34809/ using document root <none> and password file <none>
I20260927 23:09:28.530488 26775 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:28.530566 26775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:28.530810 26775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:28.533480 26775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/data/instance:
uuid: "2079e781fcca4811b957282ad7531913"
format_stamp: "Formatted at 2026-09-27 23:09:28 on dist-test-slave-lmfp"
server_key: "387d25a17e235ef3c925612839dc198e"
server_key_iv: "ca7b8e96c4b06371191ac085b0feaa7a"
server_key_version: "encryptionkey@0"
I20260927 23:09:28.534164 26775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2079e781fcca4811b957282ad7531913"
format_stamp: "Formatted at 2026-09-27 23:09:28 on dist-test-slave-lmfp"
server_key: "387d25a17e235ef3c925612839dc198e"
server_key_iv: "ca7b8e96c4b06371191ac085b0feaa7a"
server_key_version: "encryptionkey@0"
I20260927 23:09:28.538837 26775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260927 23:09:28.542079 26790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:28.543478 26775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260927 23:09:28.543660 26775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/wal
uuid: "2079e781fcca4811b957282ad7531913"
format_stamp: "Formatted at 2026-09-27 23:09:28 on dist-test-slave-lmfp"
server_key: "387d25a17e235ef3c925612839dc198e"
server_key_iv: "ca7b8e96c4b06371191ac085b0feaa7a"
server_key_version: "encryptionkey@0"
I20260927 23:09:28.543874 26775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:28.575469 26775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:28.579236 26775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:28.579509 26775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:28.597723 26775 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:45377
I20260927 23:09:28.597746 26841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:45377 every 8 connection(s)
I20260927 23:09:28.599123 26775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/data/info.pb
I20260927 23:09:28.602758 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:28.606842 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26775
I20260927 23:09:28.607050 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-0/wal/instance
I20260927 23:09:28.607575 4035 external_mini_cluster.cc:1428] Setting key 12570f8b540974d9e30f4b0213f633a4
I20260927 23:09:28.610261 26842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:28 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550568, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.253@KRBTEST.COM; defaulting to no policy
I20260927 23:09:28.635605 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.614984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50442 (local address 127.3.240.254:45377)
0927 23:09:28.618483 (+ 3499us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:28.618493 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:28.618530 (+ 37us) server_negotiation.cc:407] Connection header received
0927 23:09:28.618715 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:28.618726 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:28.619103 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:28.619282 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:28.620839 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:28.622033 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:28.624225 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:28.624799 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:28.624927 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:28.628685 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:28.628693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:28.628703 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:28.628732 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:28.631487 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:28.632154 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:28.632160 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:28.632162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:28.632295 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:28.632677 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:28.632682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:28.632685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:28.633115 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:28.633335 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:28.633905 (+ 570us) server_negotiation.cc:299] Negotiation successful
0927 23:09:28.634192 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3294,"thread_start_us":3176,"threads_started":1}
I20260927 23:09:28.635639 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.613905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45377 (local address 127.0.0.1:50442)
0927 23:09:28.614713 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.614762 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:28.617634 (+ 2872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:28.619357 (+ 1723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:28.619366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:28.620144 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:28.620602 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:28.620626 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:28.622187 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:28.622194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:28.624086 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:28.624107 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:28.625028 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:28.625037 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:28.625260 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:28.625922 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:28.625959 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:28.628473 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:28.631629 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:28.631639 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:28.631649 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:28.632010 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:28.632415 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:28.632421 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:28.632424 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:28.632567 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:28.633261 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:28.633272 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:28.633480 (+ 208us) client_negotiation.cc:769] Sending connection context
0927 23:09:28.633756 (+ 276us) client_negotiation.cc:240] Negotiation successful
0927 23:09:28.634341 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":146,"thread_start_us":304,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.253@KRBTEST.COM" created.
I20260927 23:09:28.638782 26842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:09:28.639972 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.639580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:48978)
0927 23:09:28.639757 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.639851 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260927 23:09:28.640318 26794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
I20260927 23:09:28.641749 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.641421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:48982)
0927 23:09:28.641547 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.641608 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 23:09:28.641987 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:41399: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
I20260927 23:09:28.664158 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:41399
--webserver_interface=127.3.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45377,127.3.240.253:41399,127.3.240.252:45507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:09:28.691311 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } attempt: 1
I20260927 23:09:28.692438 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.692011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:48998)
0927 23:09:28.692201 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.692316 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 23:09:28.693535 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.693179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49000)
0927 23:09:28.693338 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.693409 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260927 23:09:28.693825 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:41399: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
I20260927 23:09:28.762115 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } attempt: 2
I20260927 23:09:28.763341 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.762794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49010)
0927 23:09:28.763082 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.763188 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 23:09:28.764343 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.763999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49024)
0927 23:09:28.764160 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.764223 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 23:09:28.764562 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:41399: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
I20260927 23:09:28.855805 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } attempt: 3
I20260927 23:09:28.857105 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.856462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49028)
0927 23:09:28.856759 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.856888 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260927 23:09:28.858270 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:28.857772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49032)
0927 23:09:28.857963 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:28.858077 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260927 23:09:28.858487 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:41399: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
W20260927 23:09:28.975979 26849 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 23:09:28.976249 26849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:28.976310 26849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:28.976382 26849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:28.982975 26849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:28.983095 26849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:28.983156 26849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:28.983204 26849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:28.983258 26849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:28.991902 26849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45377,127.3.240.253:41399,127.3.240.252:45507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:28.993021 26849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:28.994240 26849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260927 23:09:29.001717 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } attempt: 4
I20260927 23:09:29.002903 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.002374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49044)
0927 23:09:29.002664 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.002768 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260927 23:09:29.004117 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.003626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49050)
0927 23:09:29.003868 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.003962 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20260927 23:09:29.004386 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:41399: Network error: Client connection negotiation failed: client connection to 127.3.240.253:41399: connect: Connection refused (error 111)
W20260927 23:09:29.005563 26854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:29.005607 26855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:29.005713 26857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:29.006603 26849 server_base.cc:1034] running on GCE node
I20260927 23:09:29.007387 26849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:29.008602 26849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:29.009843 26849 hybrid_clock.cc:648] HybridClock initialized: now 1790550569009782 us; error 54 us; skew 500 ppm
I20260927 23:09:29.010142 26849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550569, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:29.013063 26849 init.cc:377] Logged in from keytab as kudu/127.3.240.253@KRBTEST.COM (short username kudu)
I20260927 23:09:29.014324 26849 webserver.cc:460] Webserver started at http://127.3.240.253:33223/ using document root <none> and password file <none>
I20260927 23:09:29.014967 26849 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:29.015139 26849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:29.015441 26849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:29.018273 26849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/data/instance:
uuid: "e0b22d92c50f4a48bc368e433bc4aa4c"
format_stamp: "Formatted at 2026-09-27 23:09:29 on dist-test-slave-lmfp"
server_key: "778f91a4da80cf72b6c7cfb25dc1c4bb"
server_key_iv: "5255b4e9964639c7b0487665e56ac85a"
server_key_version: "encryptionkey@0"
I20260927 23:09:29.019203 26849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/wal/instance:
uuid: "e0b22d92c50f4a48bc368e433bc4aa4c"
format_stamp: "Formatted at 2026-09-27 23:09:29 on dist-test-slave-lmfp"
server_key: "778f91a4da80cf72b6c7cfb25dc1c4bb"
server_key_iv: "5255b4e9964639c7b0487665e56ac85a"
server_key_version: "encryptionkey@0"
I20260927 23:09:29.024073 26849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260927 23:09:29.027318 26864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:29.028632 26849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260927 23:09:29.028831 26849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/wal
uuid: "e0b22d92c50f4a48bc368e433bc4aa4c"
format_stamp: "Formatted at 2026-09-27 23:09:29 on dist-test-slave-lmfp"
server_key: "778f91a4da80cf72b6c7cfb25dc1c4bb"
server_key_iv: "5255b4e9964639c7b0487665e56ac85a"
server_key_version: "encryptionkey@0"
I20260927 23:09:29.029036 26849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:29.049721 26849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:29.053383 26849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:29.053648 26849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:29.071014 26849 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:41399
I20260927 23:09:29.071110 26915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:41399 every 8 connection(s)
I20260927 23:09:29.072300 26849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/data/info.pb
I20260927 23:09:29.074502 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26849
I20260927 23:09:29.074697 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-1/wal/instance
I20260927 23:09:29.075277 4035 external_mini_cluster.cc:1428] Setting key 5da5bb8ef0aae5589cede59877ebee91
I20260927 23:09:29.076565 26916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:29.086086 26916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550569, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.252@KRBTEST.COM" created.
I20260927 23:09:29.110067 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.088714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50456 (local address 127.3.240.254:45377)
0927 23:09:29.088924 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.088931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.090024 (+ 1093us) server_negotiation.cc:407] Connection header received
0927 23:09:29.092823 (+ 2799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.092827 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.092897 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.092987 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.094903 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.095653 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.098098 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.098538 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.098678 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.102703 (+ 4025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.102708 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.102711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.102741 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.106083 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.107150 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.107156 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.107159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.107388 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.107960 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.107965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.107968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.108276 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.108418 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.109377 (+ 959us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.109629 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:09:29.110991 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.089084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45377 (local address 127.0.0.1:50456)
0927 23:09:29.089844 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.089898 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:29.092524 (+ 2626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:29.093237 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:29.093249 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:29.094016 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:29.094740 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.094765 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.095812 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.095821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.097941 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.097957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.098796 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.098806 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.099133 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.099873 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:29.099908 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:29.102478 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:29.106228 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.106237 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.106253 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.106930 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.107566 (+ 636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.107571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.107574 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.107835 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.108405 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:29.108412 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:29.108851 (+ 439us) client_negotiation.cc:769] Sending connection context
0927 23:09:29.109225 (+ 374us) client_negotiation.cc:240] Negotiation successful
0927 23:09:29.109747 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":287,"threads_started":1}
I20260927 23:09:29.112759 26916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550569, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:09:29.129182 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.113547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49066)
0927 23:09:29.113760 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.113782 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:29.113887 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:29.114865 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:29.114875 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:29.115301 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:29.115433 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.115441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.117056 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.117075 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.118744 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.118764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.119768 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.119781 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.120028 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.120710 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:29.120742 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:29.123440 (+ 2698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:29.126599 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.126605 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.126610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.126987 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.127418 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.127424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.127427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.127571 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.128164 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:29.128174 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:29.128307 (+ 133us) client_negotiation.cc:769] Sending connection context
0927 23:09:29.128454 (+ 147us) client_negotiation.cc:240] Negotiation successful
0927 23:09:29.128722 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260927 23:09:29.129233 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.113724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49066 (local address 127.3.240.253:41399)
0927 23:09:29.114407 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.114412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.114437 (+ 25us) server_negotiation.cc:407] Connection header received
0927 23:09:29.114495 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.114501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.114679 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.114851 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.115642 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.116864 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.118916 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.119599 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.119721 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.123650 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.123668 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.123682 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.123714 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.126457 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.127182 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.127192 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.127196 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.127313 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.127685 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.127690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.127693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.128028 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.128259 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.128608 (+ 349us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.128840 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":358,"threads_started":1}
I20260927 23:09:29.132018 26916 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:09:29.133409 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.132819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49620)
0927 23:09:29.133048 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.133176 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260927 23:09:29.133638 26865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
I20260927 23:09:29.135305 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.134882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49626)
0927 23:09:29.135106 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.135201 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 23:09:29.135545 26916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
I20260927 23:09:29.139632 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-task9OvN8C/build/asan/bin/kudu
/tmp/dist-test-task9OvN8C/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:45507
--webserver_interface=127.3.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45377,127.3.240.253:41399,127.3.240.252:45507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260927 23:09:29.184861 26916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 1
I20260927 23:09:29.186014 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.185583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49642)
0927 23:09:29.185773 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.185871 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260927 23:09:29.187141 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.186719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49658)
0927 23:09:29.186917 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.187004 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260927 23:09:29.187363 26916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
I20260927 23:09:29.255574 26916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 2
I20260927 23:09:29.256858 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.256391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49672)
0927 23:09:29.256603 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.256702 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260927 23:09:29.257925 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.257530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49676)
0927 23:09:29.257685 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.257783 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260927 23:09:29.258117 26916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
I20260927 23:09:29.303658 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } attempt: 5
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550568, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:09:29.317966 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.304319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49080)
0927 23:09:29.304675 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.304692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:29.304792 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:29.305231 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:29.305234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:29.305660 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:29.305838 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.305846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.307277 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.307280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.308162 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.308177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.309195 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.309220 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.309409 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.310056 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:29.310072 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:29.311942 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:29.315958 (+ 4016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.315961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.315964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.316282 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.316620 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.316622 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.316624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.316740 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.317285 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:29.317288 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:29.317412 (+ 124us) client_negotiation.cc:769] Sending connection context
0927 23:09:29.317529 (+ 117us) client_negotiation.cc:240] Negotiation successful
0927 23:09:29.317718 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209}
I20260927 23:09:29.318190 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.304508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49080 (local address 127.3.240.253:41399)
0927 23:09:29.304745 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.304751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.304775 (+ 24us) server_negotiation.cc:407] Connection header received
0927 23:09:29.304917 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.304922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.305031 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.305168 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.305998 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.307147 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.308309 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.309011 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.309128 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.312118 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.312127 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.312134 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.312176 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.315842 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.316386 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.316391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.316394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.316522 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.316828 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.316832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.316835 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.317180 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.317330 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.317665 (+ 335us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.317884 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260927 23:09:29.319451 26842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:09:29.320479 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.320126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49688)
0927 23:09:29.320299 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.320363 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260927 23:09:29.321319 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.321023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49700)
0927 23:09:29.321172 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.321232 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 23:09:29.321532 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
I20260927 23:09:29.349316 26916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 3
I20260927 23:09:29.350356 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.349907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49708)
0927 23:09:29.350129 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.350211 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260927 23:09:29.351393 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.350952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49722)
0927 23:09:29.351199 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.351271 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260927 23:09:29.351590 26916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
I20260927 23:09:29.372722 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 1
I20260927 23:09:29.373787 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.373263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49730)
0927 23:09:29.373482 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.373577 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260927 23:09:29.374799 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.374403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49736)
0927 23:09:29.374564 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.374669 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 23:09:29.375073 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
I20260927 23:09:29.443305 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 2
I20260927 23:09:29.444525 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.443923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49750)
0927 23:09:29.444187 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.444292 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260927 23:09:29.445555 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.445188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49764)
0927 23:09:29.445344 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.445429 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260927 23:09:29.445804 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
W20260927 23:09:29.455341 26923 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260927 23:09:29.455664 26923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260927 23:09:29.455776 26923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260927 23:09:29.455899 26923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260927 23:09:29.462731 26923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260927 23:09:29.462805 26923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260927 23:09:29.462919 26923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260927 23:09:29.462992 26923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260927 23:09:29.463128 26923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260927 23:09:29.472471 26923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45377,127.3.240.253:41399,127.3.240.252:45507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:45507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 27 Sep 2026 22:00:12 UTC on ccycloud.quasar-sxziyj.root.comops.site
build id 4676390
ASAN enabled
I20260927 23:09:29.473707 26923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260927 23:09:29.475008 26923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260927 23:09:29.486292 26929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:29.486370 26931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260927 23:09:29.486370 26928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260927 23:09:29.486675 26923 server_base.cc:1034] running on GCE node
I20260927 23:09:29.487644 26923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260927 23:09:29.488854 26923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260927 23:09:29.490106 26923 hybrid_clock.cc:648] HybridClock initialized: now 1790550569490028 us; error 53 us; skew 500 ppm
I20260927 23:09:29.490438 26923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550569, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260927 23:09:29.493616 26923 init.cc:377] Logged in from keytab as kudu/127.3.240.252@KRBTEST.COM (short username kudu)
I20260927 23:09:29.494793 26916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 4
I20260927 23:09:29.495119 26923 webserver.cc:460] Webserver started at http://127.3.240.252:38717/ using document root <none> and password file <none>
I20260927 23:09:29.495774 26923 fs_manager.cc:359] Metadata directory not provided
I20260927 23:09:29.495903 26923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260927 23:09:29.495893 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.495421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49776)
0927 23:09:29.495634 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.495721 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260927 23:09:29.496291 26923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260927 23:09:29.496835 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.496533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49786)
0927 23:09:29.496684 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.496742 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260927 23:09:29.497053 26916 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
I20260927 23:09:29.499661 26923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/data/instance:
uuid: "233388b210a249ae865dbd172feabaf0"
format_stamp: "Formatted at 2026-09-27 23:09:29 on dist-test-slave-lmfp"
server_key: "61dc813b305b86e8f2e3926caec19265"
server_key_iv: "9b6af0eb6dcfeea273e681e89b149c39"
server_key_version: "encryptionkey@0"
I20260927 23:09:29.500414 26923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/wal/instance:
uuid: "233388b210a249ae865dbd172feabaf0"
format_stamp: "Formatted at 2026-09-27 23:09:29 on dist-test-slave-lmfp"
server_key: "61dc813b305b86e8f2e3926caec19265"
server_key_iv: "9b6af0eb6dcfeea273e681e89b149c39"
server_key_version: "encryptionkey@0"
I20260927 23:09:29.505360 26923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260927 23:09:29.508496 26938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260927 23:09:29.509686 26923 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260927 23:09:29.509858 26923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/data,/tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/wal
uuid: "233388b210a249ae865dbd172feabaf0"
format_stamp: "Formatted at 2026-09-27 23:09:29 on dist-test-slave-lmfp"
server_key: "61dc813b305b86e8f2e3926caec19265"
server_key_iv: "9b6af0eb6dcfeea273e681e89b149c39"
server_key_version: "encryptionkey@0"
I20260927 23:09:29.510064 26923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260927 23:09:29.552019 26923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260927 23:09:29.552023 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 3
I20260927 23:09:29.553031 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.552581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49796)
0927 23:09:29.552807 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.552884 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260927 23:09:29.554104 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.553694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49810)
0927 23:09:29.553914 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.554006 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260927 23:09:29.554348 26842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:45507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:45507: connect: Connection refused (error 111)
I20260927 23:09:29.556517 26923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260927 23:09:29.556768 26923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260927 23:09:29.574376 26923 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:45507
I20260927 23:09:29.574411 26989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:45507 every 8 connection(s)
I20260927 23:09:29.575678 26923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/data/info.pb
I20260927 23:09:29.578938 26990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260927 23:09:29.582697 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-task9OvN8C/build/asan/bin/kudu as pid 26923
I20260927 23:09:29.582891 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task9OvN8C/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790550224434615-4035-0/minicluster-data/master-2/wal/instance
I20260927 23:09:29.583473 4035 external_mini_cluster.cc:1428] Setting key 4bf6ab111a71acc2d8c9b84684ebb84f
I20260927 23:09:29.586243 26990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550569, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:09:29.606492 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.585141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.3.240.252:45507)
0927 23:09:29.585886 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.585897 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.585937 (+ 40us) server_negotiation.cc:407] Connection header received
0927 23:09:29.587463 (+ 1526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.587483 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.587873 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.588238 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.589759 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.591450 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.593126 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.594147 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.594344 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.598769 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.598776 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.598786 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.598813 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.602370 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.602984 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.603018 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.603028 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.603223 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.603696 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.603701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.603704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.604295 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.604567 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.605341 (+ 774us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.605673 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":545,"thread_start_us":325,"threads_started":1}
I20260927 23:09:29.607142 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.589888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.3.240.254:45377)
0927 23:09:29.590109 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.590115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.590642 (+ 527us) server_negotiation.cc:407] Connection header received
0927 23:09:29.590821 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.590827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.590902 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.591022 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.592090 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.593117 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.595131 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.595825 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.595979 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.599663 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.599668 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.599671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.599701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.602949 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.603681 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.603686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.603689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.603835 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.604209 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.604213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.604217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.604540 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.604696 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.606560 (+ 1864us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.606775 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260927 23:09:29.607148 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.589817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45377 (local address 127.0.0.1:50466)
0927 23:09:29.590492 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.590533 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:29.590680 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:29.591099 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:29.591103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:29.591658 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:29.591931 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.591940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.593288 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.593293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.594928 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.594943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.596037 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.596042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.596163 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.596740 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:29.596761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:29.599429 (+ 2668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:29.603099 (+ 3670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.603109 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.603118 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.603560 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.603949 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.603955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.603958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.604095 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.604672 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:29.604679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:29.604894 (+ 215us) client_negotiation.cc:769] Sending connection context
0927 23:09:29.605129 (+ 235us) client_negotiation.cc:240] Negotiation successful
0927 23:09:29.605568 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":1,"thread_start_us":298,"threads_started":1}
I20260927 23:09:29.608995 26990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } has no permanent_uuid. Determining permanent_uuid...
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550569, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260927 23:09:29.622506 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.609734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:41399 (local address 127.0.0.1:49096)
0927 23:09:29.609900 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.609917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:29.610004 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:29.610687 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:29.610690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:29.610985 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:29.611171 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.611179 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.612174 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.612181 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.613094 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.613105 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.613776 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.613782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.614015 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.614600 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:29.614615 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:29.616923 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:29.620327 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.620332 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.620335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.620807 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.621180 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.621184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.621187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.621355 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.621796 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:29.621802 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:29.621953 (+ 151us) client_negotiation.cc:769] Sending connection context
0927 23:09:29.622054 (+ 101us) client_negotiation.cc:240] Negotiation successful
0927 23:09:29.622232 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260927 23:09:29.622726 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.610121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.3.240.253:41399)
0927 23:09:29.610380 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.610388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.610407 (+ 19us) server_negotiation.cc:407] Connection header received
0927 23:09:29.610464 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.610470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.610543 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.610653 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.611318 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.612045 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.613236 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.613631 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.613755 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.617186 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.617194 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.617202 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.617247 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.619989 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.620927 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.620931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.620933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.621052 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.621459 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.621462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.621464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.621676 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.621822 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.622173 (+ 351us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.622381 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260927 23:09:29.623754 26990 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
I20260927 23:09:29.624522 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.609600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50468 (local address 127.3.240.254:45377)
0927 23:09:29.609753 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.609760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.609814 (+ 54us) server_negotiation.cc:407] Connection header received
0927 23:09:29.609987 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.609994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.610059 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.610144 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.611527 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.612192 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.613357 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.613739 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.613865 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.618551 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.618556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.618558 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.618587 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.621719 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.622423 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.622428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.622431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.622571 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.623041 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.623046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.623049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.623340 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.623622 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.623951 (+ 329us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.624151 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550569, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260927 23:09:29.636392 26994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.624476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49838)
0927 23:09:29.624638 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.624655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:29.624734 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:29.625115 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:29.625120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:29.625521 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:29.625670 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.625680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.626665 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.626670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.627936 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.627950 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.628570 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.628575 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.628771 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.629237 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:29.629251 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:29.631110 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:29.634067 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.634073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.634078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.634549 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.634932 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.634935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.634937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.635100 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.635667 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:29.635670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:29.635783 (+ 113us) client_negotiation.cc:769] Sending connection context
0927 23:09:29.635889 (+ 106us) client_negotiation.cc:240] Negotiation successful
0927 23:09:29.636064 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260927 23:09:29.636595 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.624716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.3.240.252:45507)
0927 23:09:29.624848 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.624853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.624868 (+ 15us) server_negotiation.cc:407] Connection header received
0927 23:09:29.624912 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.624916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.624976 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.625060 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.625800 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.626548 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.628060 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.628423 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.628554 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.631277 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.631283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.631294 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.631320 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.633945 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.634681 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.634686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.634689 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.634825 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.635208 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.635215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.635219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.635553 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.635704 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.636018 (+ 314us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.636244 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260927 23:09:29.640224 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.626171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.3.240.253:41399)
0927 23:09:29.626363 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.626368 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.626382 (+ 14us) server_negotiation.cc:407] Connection header received
0927 23:09:29.626474 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.626495 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.626556 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.626652 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.628395 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.629004 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.630103 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.630450 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.630595 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.634717 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.634725 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.634729 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.634767 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.637765 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.638407 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.638412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.638416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.638550 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.638912 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.638918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.638921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.639230 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.639484 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.639733 (+ 249us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.639928 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260927 23:09:29.641140 26990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: Bootstrap starting.
I20260927 23:09:29.644553 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260927 23:09:29.644989 26990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:29.646179 26990 log.cc:824] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:29.649294 26990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: No bootstrap required, opened a new log
I20260927 23:09:29.647595 26817 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2079e781fcca4811b957282ad7531913"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50468
I20260927 23:09:29.659534 26990 raft_consensus.cc:348] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } }
I20260927 23:09:29.659894 26990 raft_consensus.cc:729] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233388b210a249ae865dbd172feabaf0, State: Initialized, Role: FOLLOWER
I20260927 23:09:29.660362 26990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } }
I20260927 23:09:29.661190 26996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:29.661496 26996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:29.662292 26990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [sys.catalog]: configured and running, proceeding with master startup.
W20260927 23:09:29.670722 27007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260927 23:09:29.670826 27007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260927 23:09:29.669055 26923 master_runner.cc:186] Error getting master registration for 127.3.240.254:45377: OK, instance_id { permanent_uuid: "2079e781fcca4811b957282ad7531913" instance_seqno: 1790550568592556 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 23:09:29.676506 26923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 23:09:29.731560 26842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 4
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550568, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260927 23:09:29.748134 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.732383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.3.240.252:45507)
0927 23:09:29.732964 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.732969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.732989 (+ 20us) server_negotiation.cc:407] Connection header received
0927 23:09:29.733058 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.733065 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.733137 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.733237 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.734130 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.735316 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.737011 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.737651 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.737780 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.741615 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.741621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.741626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.741665 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.745171 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.745999 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.746004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.746007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.746142 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.746555 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.746560 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.746564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.746985 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.747169 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.747535 (+ 366us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.747764 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260927 23:09:29.748137 26844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.732311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49842)
0927 23:09:29.732513 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.732536 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:29.732689 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:29.733281 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:29.733286 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:29.733705 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:29.733965 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.733978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.735489 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.735496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.736876 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.736889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.737796 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.737803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.737997 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.738741 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:29.738767 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:29.741429 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:29.745301 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.745304 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.745307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.745887 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.746245 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.746251 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.746255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.746434 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.747107 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:29.747111 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:29.747279 (+ 168us) client_negotiation.cc:769] Sending connection context
0927 23:09:29.747430 (+ 151us) client_negotiation.cc:240] Negotiation successful
0927 23:09:29.747699 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260927 23:09:29.753758 26842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913: Bootstrap starting.
I20260927 23:09:29.757688 26842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:29.758932 26842 log.cc:824] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:29.761773 26842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913: No bootstrap required, opened a new log
I20260927 23:09:29.768879 26842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } }
I20260927 23:09:29.769270 26842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2079e781fcca4811b957282ad7531913, State: Initialized, Role: FOLLOWER
I20260927 23:09:29.769677 26842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } }
I20260927 23:09:29.770543 27010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:29.770963 27010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:29.771649 26842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:29.771740 26817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260927 23:09:29.771929 26817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:29.775161 26817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } }
I20260927 23:09:29.776680 26817 leader_election.cc:288] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [CANDIDATE]: Term 1 election: Requested vote from peers e0b22d92c50f4a48bc368e433bc4aa4c (127.3.240.253:41399), 233388b210a249ae865dbd172feabaf0 (127.3.240.252:45507)
I20260927 23:09:29.777501 26965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2079e781fcca4811b957282ad7531913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "233388b210a249ae865dbd172feabaf0"
I20260927 23:09:29.778079 26965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:29.782414 26965 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2079e781fcca4811b957282ad7531913 in term 1.
I20260927 23:09:29.783100 26791 leader_election.cc:302] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2079e781fcca4811b957282ad7531913, 233388b210a249ae865dbd172feabaf0; no voters:
I20260927 23:09:29.783557 27010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [term 1 FOLLOWER]: Leader election won for term 1
I20260927 23:09:29.785681 27010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [term 1 LEADER]: Becoming Leader. State: Replica: 2079e781fcca4811b957282ad7531913, State: Running, Role: LEADER
I20260927 23:09:29.786274 27010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } }
I20260927 23:09:29.796231 26916 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } attempt: 5
I20260927 23:09:29.798669 27019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2079e781fcca4811b957282ad7531913. Latest consensus state: current_term: 1 leader_uuid: "2079e781fcca4811b957282ad7531913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:29.799055 27019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: This master's current role is: LEADER
Sep 27 23:09:29 dist-test-slave-lmfp krb5kdc[26762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790550569, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
W20260927 23:09:29.808828 26944 tablet.cc:2369] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:09:29.819008 27025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260927 23:09:29.820706 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.796818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:45507 (local address 127.0.0.1:49852)
0927 23:09:29.797064 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0927 23:09:29.797082 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0927 23:09:29.797217 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0927 23:09:29.797566 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0927 23:09:29.797570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0927 23:09:29.798107 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0927 23:09:29.798330 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.798341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.799955 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.799960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.801351 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0927 23:09:29.801364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.802385 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.802391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0927 23:09:29.802652 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.803373 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0927 23:09:29.803395 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0927 23:09:29.807366 (+ 3971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0927 23:09:29.815870 (+ 8504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.815875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.815879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.816785 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.817607 (+ 822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0927 23:09:29.817637 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0927 23:09:29.817643 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0927 23:09:29.817928 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0927 23:09:29.819157 (+ 1229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0927 23:09:29.819163 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0927 23:09:29.819386 (+ 223us) client_negotiation.cc:769] Sending connection context
0927 23:09:29.819567 (+ 181us) client_negotiation.cc:240] Negotiation successful
0927 23:09:29.819983 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260927 23:09:29.823089 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.796871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.3.240.252:45507)
0927 23:09:29.797061 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.797066 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.797139 (+ 73us) server_negotiation.cc:407] Connection header received
0927 23:09:29.797343 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.797349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.797440 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.797541 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.798494 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.799805 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.801510 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.802238 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.802354 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.807687 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.807696 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.807704 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.807751 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.815637 (+ 7886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.817095 (+ 1458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.817104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.817110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.817440 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.818185 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.818193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.818198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.818853 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.821324 (+ 2471us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.821599 (+ 275us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.822167 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260927 23:09:29.828938 26916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c: Bootstrap starting.
W20260927 23:09:29.832063 26796 tablet.cc:2369] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260927 23:09:29.832954 26916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c: Neither blocks nor log segments found. Creating new log.
I20260927 23:09:29.834247 26916 log.cc:824] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c: Log is configured to *not* fsync() on all Append() calls
I20260927 23:09:29.834435 27025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260927 23:09:29.849560 26916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c: No bootstrap required, opened a new log
I20260927 23:09:29.881217 26965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [term 1 FOLLOWER]: Refusing update from remote peer 2079e781fcca4811b957282ad7531913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:09:29.882246 27019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:09:29.884611 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.803391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50480 (local address 127.3.240.254:45377)
0927 23:09:29.803589 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.803598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.849591 (+ 45993us) server_negotiation.cc:407] Connection header received
0927 23:09:29.850423 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.850429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.850510 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.850582 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.856675 (+ 6093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.857856 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.863181 (+ 5325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.864173 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.864327 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.868987 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.868997 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.869005 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.869075 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.876217 (+ 7142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.878841 (+ 2624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.878846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.878849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.879066 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.880180 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.880185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.880189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.880558 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.883827 (+ 3269us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.884005 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.884253 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260927 23:09:29.886179 26916 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } }
I20260927 23:09:29.886581 26916 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0b22d92c50f4a48bc368e433bc4aa4c, State: Initialized, Role: FOLLOWER
I20260927 23:09:29.888725 26916 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } }
I20260927 23:09:29.900564 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.807787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.3.240.252:45507)
0927 23:09:29.823933 (+ 16146us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.823942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.860131 (+ 36189us) server_negotiation.cc:407] Connection header received
0927 23:09:29.860540 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.860546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.860626 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.860744 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.867218 (+ 6474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.868575 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.871738 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.872376 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.872515 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.884880 (+ 12365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.884886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.884892 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.884932 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.888530 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.889884 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.889890 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.889894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.890030 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.890478 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.890482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.890485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.890848 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.891108 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.899882 (+ 8774us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.900156 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15812,"thread_start_us":39278,"threads_started":1}
I20260927 23:09:29.903944 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:29.847865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.3.240.253:41399)
0927 23:09:29.848075 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:29.848080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:29.860911 (+ 12831us) server_negotiation.cc:407] Connection header received
0927 23:09:29.863645 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:29.863652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:29.863721 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:29.863788 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:29.875379 (+ 11591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.876411 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.878086 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:29.878762 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:29.878884 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:29.882954 (+ 4070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0927 23:09:29.882961 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0927 23:09:29.882966 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0927 23:09:29.883023 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0927 23:09:29.887809 (+ 4786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.899104 (+ 11295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.899110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.899114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.899318 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0927 23:09:29.901911 (+ 2593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0927 23:09:29.901917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0927 23:09:29.901921 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0927 23:09:29.902344 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0927 23:09:29.902580 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:29.903336 (+ 756us) server_negotiation.cc:299] Negotiation successful
0927 23:09:29.903589 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260927 23:09:29.911748 26996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2079e781fcca4811b957282ad7531913. Latest consensus state: current_term: 1 leader_uuid: "2079e781fcca4811b957282ad7531913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:29.912132 26996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:29.917342 26996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2079e781fcca4811b957282ad7531913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:29.917570 26996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0 [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:29.876214 26775 master_runner.cc:186] Error getting master registration for 127.3.240.253:41399: OK, instance_id { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" instance_seqno: 1790550569066163 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260927 23:09:29.920024 26775 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260927 23:09:29.924005 27034 mvcc.cc:205] Tried to move back new op lower bound from 7334095134222856192 to 7334095133850742784. Current Snapshot: MvccSnapshot[applied={T|T < 7334095134222856192}]
I20260927 23:09:29.923753 27037 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:29.924429 27037 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:29.928016 26916 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [sys.catalog]: configured and running, proceeding with master startup.
I20260927 23:09:29.928104 26891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2079e781fcca4811b957282ad7531913" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c"
I20260927 23:09:29.928511 26890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [term 0 FOLLOWER]: Advancing to term 1
I20260927 23:09:29.934206 26890 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [term 1 FOLLOWER]: Refusing update from remote peer 2079e781fcca4811b957282ad7531913: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260927 23:09:29.935612 27019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260927 23:09:29.936496 27010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2079e781fcca4811b957282ad7531913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:29.936904 27010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:29.937652 26891 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2079e781fcca4811b957282ad7531913 in term 1.
I20260927 23:09:29.937779 27025 catalog_manager.cc:1347] Generated new cluster ID: 7068a4a956e443399251e1bd3f6de352
I20260927 23:09:29.937891 27025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260927 23:09:29.940259 27036 mvcc.cc:205] Tried to move back new op lower bound from 7334095134222856192 to 7334095133850742784. Current Snapshot: MvccSnapshot[applied={T|T < 7334095134222856192}]
I20260927 23:09:29.965312 27025 catalog_manager.cc:1370] Generated new certificate authority record
I20260927 23:09:29.968683 27025 catalog_manager.cc:1504] Loading token signing keys...
I20260927 23:09:29.998040 27025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913: Generated new TSK 0
I20260927 23:09:29.999491 27025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260927 23:09:30.007738 27049 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2079e781fcca4811b957282ad7531913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:30.008100 27049 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:30.008265 27019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2079e781fcca4811b957282ad7531913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:30.008518 27019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2079e781fcca4811b957282ad7531913 [sys.catalog]: This master's current role is: LEADER
I20260927 23:09:30.008425 27037 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2079e781fcca4811b957282ad7531913. Latest consensus state: current_term: 1 leader_uuid: "2079e781fcca4811b957282ad7531913" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2079e781fcca4811b957282ad7531913" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45377 } } peers { permanent_uuid: "e0b22d92c50f4a48bc368e433bc4aa4c" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 41399 } } peers { permanent_uuid: "233388b210a249ae865dbd172feabaf0" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 45507 } } }
I20260927 23:09:30.008651 27037 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c [sys.catalog]: This master's current role is: FOLLOWER
I20260927 23:09:30.045883 27056 catalog_manager.cc:1259] Loaded cluster ID: 7068a4a956e443399251e1bd3f6de352
I20260927 23:09:30.045987 27056 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c: loading cluster ID for follower catalog manager: success
I20260927 23:09:30.049880 27056 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c: acquiring CA information for follower catalog manager: success
I20260927 23:09:30.052193 27056 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e0b22d92c50f4a48bc368e433bc4aa4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260927 23:09:30.161365 26845 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.149014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50488 (local address 127.3.240.254:45377)
0927 23:09:30.149235 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.149241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.158251 (+ 9010us) server_negotiation.cc:407] Connection header received
0927 23:09:30.159301 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.159310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.159443 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.159536 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.161018 (+ 1482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50488: BlockingRecv error: recv got EOF from 127.0.0.1:50488 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260927 23:09:30.168771 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.155365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.3.240.252:45507)
0927 23:09:30.155590 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.155601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.165896 (+ 10295us) server_negotiation.cc:407] Connection header received
0927 23:09:30.166382 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.166388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.166480 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.166545 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.168513 (+ 1968us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49864: BlockingRecv error: recv got EOF from 127.0.0.1:49864 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 23:09:30.169229 26920 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.154833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.3.240.253:41399)
0927 23:09:30.155069 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.155076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.167612 (+ 12536us) server_negotiation.cc:407] Connection header received
0927 23:09:30.167872 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.167880 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.167994 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.168078 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.168972 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49138: BlockingRecv error: recv got EOF from 127.0.0.1:49138 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260927 23:09:30.225708 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.210971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49142 (local address 127.3.240.253:41399)
0927 23:09:30.211189 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.211195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.218058 (+ 6863us) server_negotiation.cc:407] Connection header received
0927 23:09:30.218584 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.218591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.218830 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.218967 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.219648 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.220701 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.222707 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.223418 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.223501 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.224209 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.224915 (+ 706us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.225033 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.225266 (+ 233us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.225350 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
W20260927 23:09:30.273156 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.259717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.3.240.252:45507)
0927 23:09:30.259929 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.259936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.270850 (+ 10914us) server_negotiation.cc:407] Connection header received
0927 23:09:30.271406 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.271413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.271547 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.271628 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.272839 (+ 1211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49876: BlockingRecv error: recv got EOF from 127.0.0.1:49876 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:09:30.311956 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.304270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49158 (local address 127.3.240.253:41399)
0927 23:09:30.304468 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.304475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.305374 (+ 899us) server_negotiation.cc:407] Connection header received
0927 23:09:30.305607 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.305613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.305723 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.305830 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.306373 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.307664 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.309691 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.310441 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.310885 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.311045 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.311233 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.311336 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.311514 (+ 178us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.311611 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260927 23:09:30.333033 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.321306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.3.240.252:45507)
0927 23:09:30.321503 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.321509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.331284 (+ 9775us) server_negotiation.cc:407] Connection header received
0927 23:09:30.331482 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.331488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.331585 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.331699 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.332768 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49892: BlockingRecv error: recv got EOF from 127.0.0.1:49892 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260927 23:09:30.346091 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.338870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49166 (local address 127.3.240.253:41399)
0927 23:09:30.339098 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.339105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.339647 (+ 542us) server_negotiation.cc:407] Connection header received
0927 23:09:30.339853 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.339859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.339946 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.340061 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.340579 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.341803 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.343904 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.344572 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.344706 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.345209 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.345353 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.345451 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.345649 (+ 198us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.345741 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260927 23:09:30.358065 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.356058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.3.240.252:45507)
0927 23:09:30.356263 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.356269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.356503 (+ 234us) server_negotiation.cc:407] Connection header received
0927 23:09:30.356708 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.356715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.356808 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.356940 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.357797 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49898: BlockingRecv error: recv got EOF from 127.0.0.1:49898 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260927 23:09:30.377842 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.370885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49182 (local address 127.3.240.253:41399)
0927 23:09:30.371130 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.371136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.371565 (+ 429us) server_negotiation.cc:407] Connection header received
0927 23:09:30.371830 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.371837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.371933 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.372042 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.372548 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.373730 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.375690 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.376398 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.376544 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.376997 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.377174 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.377282 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.377444 (+ 162us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.377531 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260927 23:09:30.386684 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.384636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.3.240.252:45507)
0927 23:09:30.384868 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.384874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.385342 (+ 468us) server_negotiation.cc:407] Connection header received
0927 23:09:30.385483 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.385488 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.385578 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.385680 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.386459 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49906: BlockingRecv error: recv got EOF from 127.0.0.1:49906 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260927 23:09:30.408041 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.400737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.3.240.253:41399)
0927 23:09:30.400947 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.400954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.401473 (+ 519us) server_negotiation.cc:407] Connection header received
0927 23:09:30.401721 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.401726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.401815 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.401946 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.402502 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.403724 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.405683 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.406525 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.406689 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.407134 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.407284 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.407395 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.407594 (+ 199us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.407687 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260927 23:09:30.416723 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.414553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.3.240.252:45507)
0927 23:09:30.414740 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.414746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.415299 (+ 553us) server_negotiation.cc:407] Connection header received
0927 23:09:30.415455 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.415460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.415548 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.415642 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.416500 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49922: BlockingRecv error: recv got EOF from 127.0.0.1:49922 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260927 23:09:30.442615 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.436268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.3.240.253:41399)
0927 23:09:30.436476 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.436483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.437064 (+ 581us) server_negotiation.cc:407] Connection header received
0927 23:09:30.437277 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.437281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.437344 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.437411 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.437929 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.438695 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.440713 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.441215 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.441328 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.441917 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.442017 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.442082 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.442286 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.442350 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260927 23:09:30.451114 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.449158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49936 (local address 127.3.240.252:45507)
0927 23:09:30.449298 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.449303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.449659 (+ 356us) server_negotiation.cc:407] Connection header received
0927 23:09:30.449844 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.449848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.449916 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.449993 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.450863 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49936: BlockingRecv error: recv got EOF from 127.0.0.1:49936 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
I20260927 23:09:30.493187 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.486932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.3.240.253:41399)
0927 23:09:30.487208 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.487216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.487571 (+ 355us) server_negotiation.cc:407] Connection header received
0927 23:09:30.487749 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.487756 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.487897 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.488017 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.488387 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.489550 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.490858 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.491669 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.491852 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.492236 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.492399 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.492527 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.492711 (+ 184us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.492798 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260927 23:09:30.502105 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.500062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.3.240.252:45507)
0927 23:09:30.500261 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.500267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.500592 (+ 325us) server_negotiation.cc:407] Connection header received
0927 23:09:30.500803 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.500806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.500889 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.500967 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.501879 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49942: BlockingRecv error: recv got EOF from 127.0.0.1:49942 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260927 23:09:30.576782 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.570239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.3.240.253:41399)
0927 23:09:30.570494 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.570503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.570863 (+ 360us) server_negotiation.cc:407] Connection header received
0927 23:09:30.571065 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.571071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.571182 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.571325 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.571680 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.572815 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.574493 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.575253 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.575392 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.575851 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.576001 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.576094 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.576373 (+ 279us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.576457 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260927 23:09:30.589116 26993 negotiation.cc:337] Failed RPC negotiation. Trace:
0927 23:09:30.583829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.3.240.252:45507)
0927 23:09:30.584066 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.584075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.587582 (+ 3507us) server_negotiation.cc:407] Connection header received
0927 23:09:30.587816 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.587822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.587927 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.588063 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0927 23:09:30.588840 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49950: BlockingRecv error: recv got EOF from 127.0.0.1:49950 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260927 23:09:30.672823 27007 catalog_manager.cc:1259] Loaded cluster ID: 7068a4a956e443399251e1bd3f6de352
I20260927 23:09:30.672916 27007 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: loading cluster ID for follower catalog manager: success
I20260927 23:09:30.675292 27007 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: acquiring CA information for follower catalog manager: success
I20260927 23:09:30.677106 27007 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 233388b210a249ae865dbd172feabaf0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260927 23:09:30.726636 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.721025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49224 (local address 127.3.240.253:41399)
0927 23:09:30.721359 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.721366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.721591 (+ 225us) server_negotiation.cc:407] Connection header received
0927 23:09:30.721889 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.721894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.721963 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.722069 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.722508 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.723292 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.724918 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.725346 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.725497 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.725891 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.726005 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.726104 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.726286 (+ 182us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.726366 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260927 23:09:30.737937 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.732157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49956 (local address 127.3.240.252:45507)
0927 23:09:30.732331 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.732335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.732764 (+ 429us) server_negotiation.cc:407] Connection header received
0927 23:09:30.732879 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.732882 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.733030 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.733140 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.733508 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.734239 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.735539 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.735912 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.736006 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.736641 (+ 635us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.737270 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.737400 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.737542 (+ 142us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.737608 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260927 23:09:30.766108 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.743632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.3.240.254:45377)
0927 23:09:30.757409 (+ 13777us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.757420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.757454 (+ 34us) server_negotiation.cc:407] Connection header received
0927 23:09:30.757634 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.757644 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.757943 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.758234 (+ 291us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.759118 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.760236 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.762218 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.762970 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.763153 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.763841 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.764333 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.764465 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.765633 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.765737 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13579,"thread_start_us":14789,"threads_started":1}
I20260927 23:09:30.766824 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.743837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.3.240.253:41399)
0927 23:09:30.744140 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.744150 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.759782 (+ 15632us) server_negotiation.cc:407] Connection header received
0927 23:09:30.759912 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.759917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.760008 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.760103 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.761316 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.762387 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.764605 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.765354 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.765500 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.766003 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.766157 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.766262 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.766466 (+ 204us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.766562 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260927 23:09:30.766969 26993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0927 23:09:30.744208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.3.240.252:45507)
0927 23:09:30.744454 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0927 23:09:30.744462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0927 23:09:30.759477 (+ 15015us) server_negotiation.cc:407] Connection header received
0927 23:09:30.759893 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0927 23:09:30.759900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0927 23:09:30.759997 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0927 23:09:30.760113 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0927 23:09:30.761022 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.762146 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.764559 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0927 23:09:30.765369 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0927 23:09:30.765476 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0927 23:09:30.766201 (+ 725us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0927 23:09:30.766334 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0927 23:09:30.766417 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0927 23:09:30.766617 (+ 200us) server_negotiation.cc:299] Negotiation successful
0927 23:09:30.766709 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260927 23:09:30.771930 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26775
I20260927 23:09:30.772961 26837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:30.885687 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26775
I20260927 23:09:30.908281 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26849
I20260927 23:09:30.909437 26911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:31.013465 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26849
I20260927 23:09:31.033367 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task9OvN8C/build/asan/bin/kudu pid 26923
I20260927 23:09:31.034435 26985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260927 23:09:31.140841 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task9OvN8C/build/asan/bin/kudu with pid 26923
2026-09-27T23:09:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4309 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9073 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (346679 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST